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PREFACE 


As  a  result  of  the  1981"  Defense  Science  Board  Summer  Study 
on  Operational  Readiness,  Task  Order  T-2-126  was  generated  to 
look  at  potential  steps  toward  improving  the  Material  Readiness 
Posture  of  DoD  (Short  Title:  R&M  Study).  This  task  order  was 
structured  to  address  the  improvement  of  R&M  and  readiness 
through  innovative  program  structuring  and  applications  of  new 
and  advancing  technology.  Volume  I  summarizes  the  total  study 
activity.  Volume  II  integrates  analysis  relative  to  Volume  III 
program  structuring  aspects,  and  Volume  IV,  new  and  advancing 
technology  aspects. 

The  objective  of  this  study  as  defined  by  the  task  order 

is: 


"Identify  and  provide  support  for  high  payof f  .act ions 
which  the  DoD  can  take  to  improve  the  military  system 
design,  development  and  support  process  so  as  to  pro¬ 
vide  quantum  improvement  in  R&M  and  readiness  through 
innovative  uses  of  advancing  technology  and  program 
structure . " 

The  scope  of  this  study  as  defined  by  the  task  order  is: 

To  (1)  identify  high-payoff  areas  where  the  DoD  could 
improve  current  system  design,  development  program 
structure  and  system  support  policies,  with  the  objec¬ 
tive  of  enhancing  peacetime  availability  of  major 
weapons  systems  and  the  potential  to  make  a  rapid 
transition  to  high  wartime  activity  rates,  to  sustain 
such  rates  and  to  do  so  with  the  most  economical  use 
of  scarce  resources  possible,  (2)  assess  the  impact  of 
advancing  technology  on  the  recommended  approaches 
and  guidelines,  and  (3)  evaluate  the  potential  and 
recommend  strategies  that  might  result  in  quantum  in¬ 
creases  in  R&M  or  readiness  through  innovative  uses 
of  advancing  technology. 
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The  approach  taken  for  the  study  was  focused  on  producing 
meaningful  implementable  recommendations  substantiated  by  quan¬ 
titative  data  with  implementation  plans  and  vehicles  to  be  pro¬ 
vided  where  practical.  To  accomplish  this,  emphasis  was  placed 
upon  the  elucidation  and  integration  of  the  expert  knowledge 
and  experience  of  engineers,  developers,  managers,  testers  and 
users  involved  with  the  complete  acquisition  cycle  of  weapons 
systems  programs  as  well  as  upon  supporting  analysis.  A  search 
was  conducted  through  major  industrial  companies,  a  director 


selected 

and 

the  following  general  plan  was  adopted. 

General  Study  Plan 

Vol.  Ill 

• 

Select,  analyze  and  review  existing 
successful  program 

Vol.  IV 

• 

Analyze  and  review  related  new  and 
advanced  technology 

Vol.  II 

•  • 

Analyze  and  integrate  review  results 

Develop,  coordinate  and  refine  new  concepts 

Vol.  I 

• 

Present  new  concepts  to  DoD  with  implementa¬ 
tion  plan  and  recommendations  for  application. 

The  approach  to  implementing  the  plan  was  based  on  an 
executive  council  core  group  for  organization,  analysis,  inte¬ 
gration  and  continuity;  making  extensive  use  of  working  groups, 
heavy  military  and  industry  involvement  and  participation,  and 
coordination  and  refinement  through  joint  industry/service 
analysis  and  review.  Overall  study  organization  is  shown  in 
Fig.  P-1. 

The  basic  technology  study  approach  was  to  build  a  founda¬ 
tion  for  analysis  and  to  analyze  areas  of  technology  to  surface: 
technology  available  today  which  might  be  applied  more  broadly; 
technology  which  requires  demonstration  to  finalize  and  reduce 
risk;  and  technology  which  requires  action  today  to  provide  reli 
able  and  maintainable  systems  in  the  future.  Program  structur¬ 
ing  implications  were  also  considered.  Tools  used  to  accomplish 
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this  were  existing  documents,  reports  and  study  efforts  such  as 
the  Militarily  Critical  Technologies  List.  To  accomplish  the 
technology  studies,  sixteen  working  groups  were  formed  and  the 
organization  shown  in  Fig.  P-2  was  established. 

This  document  records  the  activities  and  findings  of  the 
Technology  Working  Group  for  the  specific  technology  as  indi¬ 
cated  in  Fig.  P-2.  The  views  expressed  within  this  document 
are  those  of  the  working  group  only.  Publication  of  this  docu¬ 
ment  does  not  indicate  endorsement  by  IDA,  its  staff,  or  its 
sponsoring  agencies. 

Without  the  detailed  efforts,  energies,  patience  and 
candidness  of  those  intimately  involved  in  the  technologies 
studied,  this  technology  study  effort  would  not  have  been 
possible  within  the  time  and  resources  available. 
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A 


FOREWORD 


The  recognition  of  the  urgent  need  for  a  strong  COD  Testing 
Technology  Program  is  not  a  new  idea.  In  a  study,  initiated  by  the 
Assistant  Secretary  of  the  Navy  for  Research  and  Development  in  1976, 
a  Testing  Technology  Program  was  defined.  The  Navy,  through  the  Naval 
Ocean  Systems  Center,  put  "meat  on  the  bones"  and,  together  with  the 
Navy  laboratories  and  other  concerned  activities,  prepared  a  Testing 
Technology  Program  Plan.  Through  the  JLC  Panel  On  Automatic  Testing, 
the  Program  was  expanded  to  include  the  needs  of  the  other  Services. 

Progress  in  implementing  this  Program  has  been  slow.  The 
enthusiasm  of  the  testing  technology  advocates  throughout  DOD  and 
industry  remains  strong.  Unfortunately,  to  a  large  extent,  this 
enthusiasm  has  not  been  effectively  transferred  to  the  many  layers  of 
management  extending  from  DOD  through  Congress. 

The  significant  potential  for  improving  the  reliability  and 
maintainability  by  investments  in  testing  technology  still  remains. 

It  is  anticipated  that,  through  this  report,  the  requirement  for  a 
strong  Testing  Technology  Program  is  "laid  to  rest"  and  support  from 
Congress,  DOD,  and  the  Services  ultimately  will  provide  a  significant 
and  balanced  Program. 

The  recommendations  in  this  report  provide  the  roadmap  for 
reaching  this  goal.  Sound  leadership  is  required  to  make  this 
happen. 


George  W.  Neumann 

Chairman,  Testing  Technology  Committee 


EXECUTIVE  SUMMARY 


Section 

1.1 

1.2 

1.3 

1.4 

1.4. 1 

1.4.2 

1.4.3 

1.5 

1.5.1 
1.5-2 

1.5.3 

1.5.4 

Sect i on 
Section 
Section 

4. 1 

4.1.1 

4.1.2 

4.1.3 

4.1.4 

4.1.5 

4.2 

4.3 

4.4 

4.5 

Section 

5.1 


5.2 

5-3 

5.4 


TABLE  OF  CONTENTS 


1  -  Testing  Technology  .  1-1 

Introduction  .  1-1 

Description  Of  Entire  Reliability  And  Maintainabi  1  ity 

Study  Program .  1-1 

Description  Of  The  Testing  Technology .  1-3 

Testing  Problems  In  The  Field .  1-3 

Weapon  System  Testability .  1-6 

Built-In-Test  (BIT) .  1-6 

Test  Equipment .  1-3 

Problems  In  Applying  Testing  Technology  During  The 

Weapon  System  Acquisition  Process .  1-8 

Advancing  Embedded  Testing  Support  Technology .  1-8 

Applying  Testing  Technology  Early  In  The  Weapon 

System  Acquisition  Cycle  .  1-11 

Transitioning  Testing  Technology  .  1-11 

Utilizing  Industry  IRsD  .  1-11 

2  -  Study  Goal  And  Objectives .  2-1 

3  -  Defining  The  Requirement .  3"1 

4  -  Testing  Technology  Requirements .  4-1 

Testing  Technology  .  4-1 

Weapon  System  Testability  Design  Techniques .  4-1 

On-Line  Testing .  4-2 

Off-Line  Testing  .  4-2 

Test  Techniques .  4-2 

Test  And  Evaluation  (Test  Beds) .  4-2 

Acquisition  Tools .  4-3 

Management . .  4-3 

Program  Funding . .  4-3 

Program  Priorities  .  4-5 

5  -  Conclusions .  5“1 

Traditional  Weapon  System  Reliability  And 
Maintainability  Design  Techniques  Are  No 

Longer  Satisfactory .  5-1 

Improvement  In  The  Technology  Is  Required .  5“1 

Injecting  Testing  Technology  Into  Weapon  Systems 

Oesigns  Must  Be  "Institutionalized" .  5-2 

The  Management  Of  Testing  Technology  Requires 

Improvement .  5-2 


EXECUTIVE  SUMMARY 
TABLE  OF  CONTENTS  (Continued) 


Section  6  -  Recommendations, 


Initiate  A  Major  Weapon  System  Design  Technology 
Program,  Which  Injects  Testing  Technology  Into 

This  Design  Process . 

Invest  In  Expanding  The  Testing  Technology  Base 
To  Provide  "Off-The-Shelf"  Proven  Alternatives 

For  Use  In  Weapon  System  Design . 

Institutionalizing  The  Transitioning  And 

Utilization  Of  Testing  Technology . 

Initiate  A  Series  Of  Actions  To  Improve 
Testing  Technology  Management  . 


A,1'*  ,v  . 

•  •  >  ---  •  ■ « *  .■  - - 


• .  • .  - ,  - .  -  .v,  * 

v.v  v.-.V-'.'- 


ure  No. 


EXECUTIVE  SUMMARY 
LIST  OF  ILLUSTRATIONS 
Title 

Reliability  and  Maintainability  Study  Program 

Structure  . 

Testing  Technology  Defined . 

Current  Problems . 

Comparison  of  Current  and  Future  Testing 

Requi remen ts . 

Embedded  Test  . 

On-Going  IR&D  Efforts  . 

Test  Technology  in  Relation  to  Weapon  System 

Design  . 

Annual  Funding  Summary  ($M)  . 


CABLING  *  STRUCTURAL 
CONNECTORS  FIBER  OPTICS  COMPOSITES 
J.  Bird  A.  Glista  F •  Crossman 


EXECUTIVE  SUMMARY 


SECTION  1.  TESTING  TECHNOLOGY 


1.1  INTRODUCTION 

This  study  report  addresses  the  requirements  for  a  testing 
technology  development  prcgram.  The  study  is  part  of  a  larger  Reliability 
and  Maintainability  Improvement  Study  Program.  The  first  portion  of  this 
report  describes  this  entire  study  and  how  testing  technology  fits  into 
its  framework.  This  is  followed  by  a  description  of  the  problem,  scope, 
goals,  objectives,  approach,  content,  payoffs,  conclusions,  and  recommenda¬ 
tions  relating  to  a  testing  technology  program. 

1.2  DESCRIPTION  OF  ENTIRE  RELIABILITY  AND  MAINTAINABILITY  STUDY  PROGRAM 

The  Under  Secretary  of  Defense  has  initiated  a  Joint  CSD-Service- 
Industry  study  for  improving  weapon  system  reliability  and  maintainability 
(R&M) .  The  objective  of  the  study  is  to  identify  and  provide  support  for 
high  payoff  actions  which  DOO  can  take  to  improve  the  military  system 
design,  development  and  support  process  in  the  areas  of  reliability  and 
maintainability,  through  innovative  uses  of  advancing  technology  and  program 
structure. 

As  shown  in  Figure  1-1,  this  Joint  OSD-Servi ce- I ndust ry  study 
program  is  divided  into  three  distinct  parts.  The  first  part  is  program 
case  studies  to  develop  a  credible  list  of  engineering,  design,  test  and 
contracting  activities,  which  when  followed  will  satisfy  the  study  objec¬ 
tives.  The  second  part  deals  wi th  effectiveness  analysis  in  order  to 
quantify  the  impact  of  R&M  investment.  The  third  part  addresses  new 
technologies  that  could  lead  to  quantum  improvements  in  R&M  and  readiness. 

This  report  addresses  the  Testing  Technology  portion  of  this  third 
part  of  the  entire  Reliability  and  Maintainability  study.  Thus,  Testing 
Technology  is  a  sub-subset  of  New  Technologies. 
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1.3  DESCRIPTION  OF  THE  TESTING  TECHNOLOGY 

Testing  technology  covers  a  gamut  of  research  and  development, 
ranging  from  basic  research  (RDT&E  category  6.1)  to  engineering  development 
(RDT&E  category  6.4).  The  technology  embraces  all  weapon  system  testing 
needs  (e.g.,  electronics,  avionics,  propulsion,  machinery)  related  to 
maintenance  of  those  systems.  As  shown  in  Figure  1-2,  it  includes  test 
equipment;  and  the  logistic  support  of  the  equipment,  which  encompasses  two 
very  expensive  items  -  test  program  sets  and  the  calibration  of  the  test 
equipment  itself.  Embedded  test  support  includes  bui 1 t- i n-test ,  readiness 
monitoring,  and  system  sel f-al ignment.  Also  included,  are  two  technologies 
which  are  inexplicably  tied  to  the  design  of  the  weapon  system.  They  are: 

1)  fault-tolerant  design  techniques,  which  when  used  in  conjunction  with 
bui 1 t-in-test ,  provide  a  very  powerful  readiness  improvement  tool;  and, 

2)  testability  design  techniques,  which  enhance  the  testing  of  units  and 
systems.  Diagnostic  and  prognostic  techniques  are  an  integral  part  of  both 

test  equipment  and  embedded  test  support. 

The  study  addresses  testing  technology  required  to  maintain  al 1 

types  of  weapon  systems.  It  does  not  include  such  testing  as  conducted 
for  reliability  and  maintainability  assurance.  Development  and  operational 
test  and  evaluation  are  also  excluded.  Although  not  specifically  addressing 
factory  testing  units  during  production,  the  integration  of  factory  and 
field  testing  can  save  significant  production,  quality  assurance,  and 

operational  testing  man  hours. 

1.4  TESTING  PROBLEMS  IN  THE  FIELD 

Testing  technology  is  both  a  readiness  and  a  life  cycle  cost  driver. 
It  has  a  significant  effect  on  most  of  the  other  ILS  elements.  It  has  a 
significant  effect  on  the  combat  readiness  and  operational  availabilit/  of 
weapon  systems  by  decreasing  Mean  Time  To  Repair  and  Mean  Logistics  Delay 
Time.  Used  in  conjunction  with  fault-tolerant  design  techniques,  testing 
technology  can  significantly  increase  the  Mean  Time  Between  Failures. 
However,  there  exist  a  number  of  problems  which  inhibit  the  effective 
application  of  this  technology.  Some  of  these  problems  are  summarized  in 
Figure  1-3,  and  discussed  below. 
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Figure  1-2.  Testing  Technology  Defined 


ion  System  Testabilit 


Weapon  systems  and  their  units,  which  have  not  been  designed  to  be 
efficiently  tested,  create  excessive  test  times,  cause  excessive  diagnostic 
and  test  programming  costs,  and  require  manpower  and  skill  levels  which  are 
not  readily  available.  Test  times  for  units  presently  fielded  which  are 
designed  to  today's  state  of  the  art,  can  run  from  a  matter  of  minutes  to 
a  number  of  hours.  Figure  1-1*  is  a  comparison  of  the  test  requirements  for 
currently  available  units  to  that  required  in  the  immediate  future.  In 
this  case,  a  unit  designed  with  VHSIC  devices  is  used.  It  is  anticipated 
that  maximum  operating  frequencies  for  these  units  will  increase  by  an 
order  of  magnitude;  the  vector  depth  (memory  per  pin)  will  increase  by 
three  orders  of  magnitude;  and,  using  the  same  testing  technology,  the 
overall  test  times  will  increase  by  three  orders  of  magnitude.  Test  pro¬ 
gram  sets  for  testing  a  complex  unit,  such  as  a  "black  box"  removed  from 
an  aircraft,  now  can  cost  over  two  million  dollars  each.  Test  program 
set  costs  for  units  built  with  VHSIC  devices  could  make  the  testing  of 
these  units  impracticable.  This  lack  of  designing  testable  weapon  systems 
and  units  has  forced  the  military  into  multi-level  maintenance  concepts, 
which  require  skilled  technicians  at  each  maintenance  level,  has  lowered 
repa’r  productivity,  and  has  resulted  in  the  need  for  extra  spares. 

1.4.2  Built-In-Test  (BIT) 

Built-in-test  for  weapon  systems,  which  is  being  introduced  into 
the  field  today,  is  not  meeting  diagnostic  specifications.  The  Air  Force 
Test  and  Evaluation  Center  (AFTEC)  in  a  study  of  three  aircraft  concluded 
that  the  diagnostic  capability  as  seen  by  the  user  was  in  the  range  of  50 
to  75. percent  and  that  the  false  alarm  rate  exceeded  85  percent  in  some 
instances.  In  a  study  conducted  by  the  Naval  Sea  Systems  Command  it  was 
found  that  70  percent  of  the  modules  removed  from  a  weapon  system  were 
eventually  found  to  be  failure-free.  Situations  such  as  these  cause  the 
technicians  to  lose  faith  in  the  operation  of  BIT  and  causes  the  logistics 
system  to  operate  inefficiently. 
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Figure  1-4.  Comparison  of  Current  and  Future  Testing  Requirements 
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|  Historically  the  testing  problem  has  been  satisfied  by  providing 

both  automatic  and  manual  test  equipment  as  the  key  maintenance  and  repair 
tool.  This  has  resulted  in  the  Services  fielding  three  million  units  of 
manual  test  equipment  that  must  be  acquired,  deployed,  and  logistically 
1  supported.  In  addition,  the  Services  have  procured  and  deployed  approxi¬ 

mately  1,000  different  types  of  ATE's,  many  costing  more  than  a  million 
dollars  each.  Most  of  the  manual  test  equipment  and  many  of  the  building 
blocks,  which  make  up  the  ATE's,  are  commercial  units  which  often  times 
i  cannot  be  logistically  supported  adequately.  Training  technicians  to  use 

i 

this  wide  variety  of  test  equipment  is  very  difficult  and  time  consuming. 

1.5  PROBLEMS  1H  APPLY  I  MG  TESTING  TECHNOLOGY  PUR  I  MG  THE  WEAPON  SYSTEM 
ACQUISITION  PROCESS 

There  are  a  number  of  problems  in  applying  testing  technology 
during  the  weapon  system  acquisition  process,  which  result  in  excessive 
life  cycle  costs.  Some  of  these  are  described  in  the  following  paragraphs. 

1.5.1  Advancing  Embedded  Testing  Support  Technology 

In  the  past,  the  emphasis  on  satisfying  testing  requirements  has 
been  placed  on  providing  more  and  better  test  equipment.  In  most  cases, 
the  Services  and  industry  have  successfully  achieved  many  advances.  However, 
the  same  emphasis  has  not  been  placed  on  embedded  testing  support.  Embedded 
testing  support  holds  promise  of  reduced  costs  to  achieve  a  given  operational 
availability.  Quite  simply,  creating  the  environment  a  weapon  system  "sees" 
in  operation  is  difficult  and  costly  to  reproduce  in  a  maintenance  shop. 

As  depicted  in  Figure  1-5,  there  is  a  mix  of  embedded  testing  and  test 
equipment  which  will  optimally  satisfy  a  given  operational  availability. 

This  mix  is  not  often  achieved. 

Each  Service  has  a  major  research  and  development  in  test  equipment 
technology.  In  addition,  industry's  IR&D  effort  is  centered  on  test  equip¬ 
ment  hardware  and  software  simply  because  this  is  what  they  can  market. 
Industry  cannot  readily  market  embedded  testing  support  and  so  there  is 
little  IR&O  in  this  area.  Figure  1-6  graphically  depicts  the  effort  which 
more  than  30  companies  sponsor  over  a  hundred  different  projects,  probably 
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with  a  dollar  value  of  10  to  20  million  dollars  per  year.  Only  a  fraction 
of  these  dollars  are  spent  on  embedded  testing  support  technology.  Embedded 
testing  support  requires  development  of  "off-the-shelf"  technology,  which  is 
ready  to  apply  at  the  early  stages  of  weapon  systems  design,  as  a  rigorous 
des i gn  discipline. 

1.5.2  Applying  Testing  Technology  Early  In  The  Weapon  System 

Acquisition  Cycle 

Historically,  prime  system  designers  have  generally  regarded 
maintenance  (including  testing)  and  logistic  support  as  an  after-design 
concept.  In  addition,  prime  contractors  do  not  adequately  and  properly 
communicate  to  subcontractors  the  scope  and  depth  of  testability  required. 
Neither  is  there  an  adequate  acceptance  test  program  to  evaluate  the  degree 
of  subcontractor  testability  conformance.  As  a  result,  the  Services  have 
been  playing  catch-up  while  many  of  their  weapon  systems  are  down.  Testing 
technology  must  be  an  integral  part  of  weapon  system  design.  As  depicted 
in  Figure  1-7,  this  technology  is  closely  tied  to  computer-aided  design, 
logistic  support  analysis  and  the  automation  of  diagnostics,  test  and 
maintenance.  Without  this  "front-end"  attention,  investments  in  testing 
technology  can  provide  only  marginal  returns  in  readiness  and  cost  reduction 

1.5.3  Transitioning  Testing  Technology 


Traditionally,  transitioning  technology  in  any  field  has  been 
difficult.  Transitioning  technology  from  basic  research  to  exploratory 
development  to  advanced  development  and  to  engineering  development  causes 
problems,  because  often  each  of  these  BDTSE  categories  is  managed  by  a 
separate  organizational  entity.  For  the  same  reason,  it  has  proved  diffi¬ 
cult  to  transition  testing  technology  to  weapon  system  design. 

US.4*  Utilizing  Industry  1R&D 

Industry's  1 RsD  investment  in  testing  technology  is  significant. 

OSD  and  the  Services  have  emphasized  the  need  for  additional  IRSD  in  each 
one  of  the  logistic  support  elements.  Without  better  incentives,  improved 
coordination,  and  follow-on  visibility,  much  of  this  industry  effort  remains 
unused. 
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Figure  1-7.  Test  Technology  in  Relation  to  Weapon  System  De 


SECTION  2.  STUDY  GOAL  AND  OBJECTIVES 


The  objectives  of  this  Testing  Technology  Study  are  to: 

o  Identify  the  required  technology  development. 

o  Estimate  the  impacts  of  thest  technology  developments. 

o  Identify  the  key  management  actions  required  to  support 
the  development  and  application  of  this  technology. 

Prepare  the  detailed  analysis  necessary  to  justify  and 
defend  the  priorities  that  must  be  afforded  this 
technology  and  the  expected  payoffs. 


o 


SECTION  3-  DEFINING  THE  REQUIREMENT 


Over  the  past  few  years  the  Services  and  industry  have  taken  a 
number  of  significant  steps  toward  defining  the  requirements  for  testing 
technol ogy  . 

The  Navy's  program  was  established  as  a  result  of  a  study  directed 
by  the  Assistant  Secretary  of  the  Navy  for  Research  and  Development 
1976.  This  study  was  culminated  in  a  "Report  On  Navy  Issues  Concerning 
Automatic  Test,  Monitoring,  And  Diagnostic  Systems  And  Equipment".  This 
Report  identified  20  basic  Fleet  problems  in  automatic  testing  and  proposed 
14  solutions.  The  Navy's  Testing  Technology  Program  today  is  based  on  the 
findings  of  this  study. 

Two  or  three  years  later,  the  Air  Force  initiated  the  Modular 
ATE  (MATE)  Program.  This  was  a  major  concept  definition  program  in 
competition  between  Sperry  and  West inghouse.  One  of  the  outputs  of  this 
conceptual  effort  was  a  set  of  MATE  guides  dealing  with  virtually  every 
aspect  of  automatic  testing  and  testability.  These  guides  are  being  used 
by  the  Air  Force  in  the  acquisition  of  their  automatic  testing  hardware  and 
software. 

The  Army  has  just  concluded,  within  the  past  year,  a  DATAT1  study, 
which  resulted  in  22  findings  which  addressed  all  aspects  of  test, 
measurement,  and  diagnostic  equipment.  This  study  has  formed  the  basis  for 
the  Army's  tec.molog i cal  and  managerial  approach  to  solving  testing  problems. 

In  1981  the  Assistant  Secretary  of  Defense,  Manpower,  Reserve 
Affai  rs  and  Logistics  sponsored  a  Bui  1 1- In-Test  Equipment  Requirements 
Workshop.  This  Workshop  was  held  for  the  purpose  of  assessing  progress 
and  problems  in  specifying  and  evaluating  bui 1 t- i n-test  used  in  complex 
electronic  equipment.  A  number  of  significant  recommendations  resulted 
from  this  Workshop.  These  recommendations  are  documented  in  the  Institute 
for  Defense  Analysis  Paper,  P-1600.  Fourteen  of  these  recommendations 
dealt  with  specifying  and  evaluating  diagnostics,  including  built-in-test. 
Another  set  of  recommendations  were  made,  which  clearly  identified  the 
need  for  technology  development  in  bui 1 t- i n-test  and  diagnostic  techniques. 
This  study  formed  the  basis  for  a  DOD-wide  program  to  improve  built-in¬ 
test  and  diagnostics. 


1  Department  of  the  Army  Test,  Measurement  and  Diagnostic  Equipment  Action  Team. 


Defining  and  coordinating  testing  technology  effort  among  the 

Services  is  being  accomplished  through  the  framework  of  the  Joint  Logistics 

Commanders  (JLC) .  The  JLC  Panel  On  Automatic  Testing  was  formed  in  1?78 

and  coordinates  and  guides  the  Joint  Services  Automatic  Testing  program. 

One  of  the  useful  testing  technology  assessments  was  developed  under  this 

Joint  Service  Automatic  Testing  program.  It  assesses  future  testing 
2 

technology  needs  .  This  report  evaluates  the  impact  of  new  technologies  on 
testing  technology  requirements.  It  also  determines  the  applicability  of 
these  new  technologies  to  solving  test  problems.  The  report  covers  new 
technology  in  systems,  components,  electromagnetic  transmissions,  computers, 
electro-optics,  and  acoustics. 

Testability,  as  a  defined  discipline,  has  been  in  being  for  the 
last  five  or  six  years.  However,  the  institutionalization  of  testability, 
including  the  ability  to  invoke  testability  requirements  in  our  weapon 
systems  designs  to  assure  that  requirements  are  met,  is  in  its  infancy. 
Through  a  Bui  1 1- In-Test/Testabi 1 i ty  Improvement  Program,  initiated 
under  the  Joint  Service  program,  the  path  for  institutionalizing 
testability  through  a  series  of  standardization  documents,  which  are  closely 
tied  to  the  logistics  support  analysis  process,  has  been  defined.  In 
addition,  RsD  has  been  recommended  to  develop,  modify,  and  evaluate  a  series 
of  testability  analytical  tools  to  aid  designers  in  performing  testability 
trade-offs. 

On  the  other  hand,  industry  has  played  an  important  part  in  defining 
the  Services'  testing  technology  program.  Two  comprehensive  studies  have 
been  supported  by  five  industry  associations:  The  Aerospace  Industries 
Association;  The  Electronic  Industries  Association;  The  National  Security 
Industrial  Association;  The  Shipbuilders  Council  Of  America;  and,  The 
American  Electronics  Association.  The  first  of  these  studies  culminated 
in  a  "Report  Of  Industry  Ad  Hoc  Automatic  Test  Equipment  Project  For  The 
Navy".  This  study  was  directed  almost  totally  at  defining  RDTsE  needs 
in  testing  technology  and  the  institutionalization  of  their  use.  A  similar 


2  NOSC  TD  426,  Technology  Assessment,  1980,  Forecast  Of  Future  Test 
Technology  Requirements  (March,  1981). 


•\  -*.v 


•■-.A.-.'-. 


type  project  for  the  Joint  Services  was  subsequently  undertaken  by  these 
five  industrial  organizations.  The  Final  Report  of  this  "Industry/Joint 
Services  Automatic  Test  Project"  addressed  the  entire  spectrum  of  automatic 
testing. 

Because  of  the  close  working  relationship  between  the  Services  and 
industry  during  this  period,  the  Services'  testing  technology  programs  are 
totally  compatible  with  the  industry  recommendations. 


SECTION  4.  TESTING  TECHNOLOGY  REQUIREMENTS 

Investing  in  testing  technology  does  little  good  if  the  Services 
do  not  provide  a  means  for  weapon  system  designers  to  easily  use  this 
technology.  A  three-pronged  approach  to  this  problem  is  indicated.  This 
includes:  l)  development  of  the  technology  itself,  coupled  with;  2)  the 
tools  used  to  apply  this  technology  in  the  weapon  system  acquisition  process 
and,  3)  appropriate  management  attention  to  ensure  proper  utilization.  Each 
of  these  three  items  is  addressed  in  this  Section. 

4.1  TESTING  TECHNOLOGY 

The  composition  of  required  testing  technology  includes  the  need 
for  weapon  system  testability  design  techniques.  All  of  these  revolve 
around  the  concept  of  a  test  bed  supported  with  advanced  development  funds 
to  evaluate  combinations  of  testing  technologies,  while  being  able  to 
ascertain  the  synergistic  effects  of  each. 

4.1.1  Weapon  System  Testability  Design  Techniques 

Testability  is  defined  as  a  design  characteristic  which  allows  the 
status  (operable,  inoperable,  or  degraded)  of  a  unit  (system,  subsystem, 
module,  or  component)  to  be  confidently  determined  in  a  timely  fashion. 
Testability  is  inherently  a  weapon  system  design  issue.  At  present,  the 
use  of  computerized  tools  in  the  design  of  a  weapon  system  is  not  an 
integrated  process.  The  design  of  the  weapon  system  itself  is  part  of  the 
computer-aided  design  (CAD)  process.  Logistics  support  analysis  (LSA)  for 
ILS  should  support  this  CAD  process,  with  testability  as  a  major  driver. 
However,  testability  as  a  rigorous  design  technique  is  in  its  infancy. 

Means  for  specifying,  predicting,  and  demonstrating  weapon  system  testabil 
ity  are  not  mature. 

The  reliability  of  deployed  weapon  systems  has  not  proved  satis¬ 
factory.  Traditional  reliability  approaches  are  expensive,  time  consuming 
and  not  altogether  satisfactory.  Fault-tolerant  design  techniques  mainly 
have  centered  around  restructuring  at  the  equipment  level,  which  is  costly 
and  creates  a  greater  maintenance  workload.  Present  effort  in  development 
of  fault-tolerant  design  techniques  is  fractionated  with  little  thought  on 
"institutionalizing"  its  use. 
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On-line  testing  is  defined  as  testing  a  weapon  system  or  unit  in 
its  operational  environment.  It  includes  bui 1 1- i n-test ,  bui 1 1- i n-test 
equipment,  performance  monitoring,  status  monitoring,  maintenance  aiding, 
etc.  Whether  on-line  testing  is  at  the  ship  level,  the  aircraft  level,  the 
vehicle  level  or  the  weapon  system  level,  it  involves  "des ign i ng- i n"  a 
comprehensive  testing  hardware  and  software  capability  during  the  acquisition 
process. 

4.1.3  Off-Line  Testing 

Off-line  testing  is  accomplished  by  a  combination  of  automatic  and 
manual  test  equipment,  coupled  with  the  necessary  software  for  test  program 
sets  required  to  diagnose  faulty  units.  Development  of  manual  test  equip¬ 
ment  required  by  the  Services  is  being  accomplished  by  industry,  using  IR&D 
funds.  Except  in  special  cases,  the  Services  do  not  and  should  not  invest 
their  dollars  to  develop  manual  test  equipment.  On  the  other  hand,  automatic 
test  equipment,  to  a  large  degree,  is  designed  to  the  operational  and  support 
requirements  of  the  Services.  Logistic  support  (including  calibration)  is 
required  for  all  types  of  test  equipment. 

4.1.4  Test  Techniques 

The  extensive,  and  sometimes  unnecessary,  maintenance  actions  on 
weapon  systems  place  high  demands  on  personnel  and  test  equipment,  and 
adversely  affect  combat  readiness.  Furthermore,  the  employment  of  new 
and  emerging  technologies,  which  offer  opportunities  for  reducing  manning 
requirements  for  future  weapon  system  operation,  will  impose  increased 
demands  on  maintenance  personnel.  Test  techniques  are  required  to  satisfy 
these  demands. 

4.1.5  Test  And  Evaluation  (Test  Beds) 

None  of  the  technology  development  discussed  in  the  above  paragraphs 
can  be  developed  in  a  vacuum.  Scientific  test  beds,  including  prototypes, 
need  to  be  utilized  for  use  in  evaluating  various  testing  technology  improve¬ 
ments  in  an  integrated,  realistip  operating  environment. 
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Applying  the  output  of  testing  technology  requires  the  institution¬ 
alization  of  acquisition  tools.  These  include: 

a.  Preparation  of  design  and  application  guides, 
standards  and  specifications  for  weapon  system 
designers. 

b.  Establishment  and  maintenance  of  i nfctrmat ional 
data  banks  for  use  with  analytical  models. 

c.  Development  and  offering  of  educational  courses 
for  project  managers  and  weapon  system  designers 
in  the  application  of  testing  technology. 

4.3  MANAGEMENT 

Transitioning  test  technology  to  weapon  system  design  requires  a 
number  of  management  initiatives. 

a.  An  organizational  entity  is  required  within  each 
Service  and  OSD  to  plan,  coordinate  and  transition 
testing  technology  through  to  weapon  system  design. 

b.  Policy  directives  are  required  for  each  Service  to 
establish  such  an  organization  and  to  assure  technology 
developments  are  funded  and  pursued. 

c.  Controls  over  development  of  testing  technology  and 
its  application  are  required. 

d.  Methods  for  Joint  Service  coordination  of  testing 
technology  is  required. 

4.4  PROGRAM  FUNDING 

Figure  4-1  is  a  funding  summary,  which  indicates  that  basic  research 
and  exploratory  development  is  funded  at  $7M  annually  -  SO  percent  of 
requirements.  Advanced  and  engineering  developments  are  funded  at  $27M  - 
77  percent  of  requirements.  This  does  not  include  a  substantial  deficit  in 
the  Navy's  Consolidated  Support  System  out-year  funding. 
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PROGRAM  PRIORITIES 


The  various  parts  of  the  Testing  Technology  Program  have  been 
prioritized  based  on  test  issues;  the  affect  on  weapon  system  operational 
readiness,  life  cycle  cost  and  manpower  considerations;  technical  risk; 
and  the  size  of  the  funding  deficiency.  High  priorities  are  given  to: 

o  Weapon  system  design,  using  tes tab i 1 i ty/B I T/faul t 
tolerance/performance  monitoring  techniques, 
incorporated  into  the  CAD/LSA  process. 

o  Oi agnost i c/prognos t ic  techniques,  integrating  FMEA, 
BIT/Testabi 1 i ty  maintenance  aiding,  ATPG  into  a 
cohesive,  institutionalized  process. 

o  Non-e lectron ic  test  and  monitoring  techniques. 

o  System-level  test  techniques. 


SECTION  5.  CONCLUSIONS 


As  a  result  of  this  study,  the  following  conclusions  have  been 

reached. 

5-1  TRADITIONAL  WEAPON  SYSTEM  RELIABILITY  AND  MAINTAINABILITY 

OESiGN  TECHNIQUES  ARE  NO  LONGER  SATISFACTORY 

Traditional  design  techniques  for  injecting  reliability  and  main¬ 
tainability  technology  into  weapon  systems  are  no  longer  satisfactory. 
Testability  and  testing  requirements  must  be  injected  into  weapon  system 
operational  requirements,  requests  for  proposals,  and  system  specifications 
beginning  at  the  weapon  systems  concept  formulation  stage  and  continuing 
throughout  the  acquisition  cycle.  These  requirements  must  be  specified 
as  "design  requirements"  and  measurable  over  the  acquisition  cycle  of  the 
weapon  system.  To  accomplish  this,  we  must  learn  to  "speak  the  language" 
of  the  weapon  system  designer.  A  "performance  over  time"  concept  must 
replace  the  "supportabi 1 i ty"  concept,  with  "performance  over  time"  equal 
in  importance  to  performance  capability.  Effectiveness  must  have  the 
relationship  between  performance  capability  and  "performance  over  time": 

E  csPc  X  Pr 

To  do  this,  we  must  learn  how  to  specify  Py.  It  must  be  mission-driven  and 
relatable  to  acquisition  and  ownership  costs. 

5.2  IMPROVEMENT  IN  THE  TECHNOLOGY  BASE  IS  REQUIRED 

The  present  technology  base  does  not  exist  to  significantly  improve 
this  situation.  Tools  do  not  exist  to  integrate  and  trade-off  various 
reliability,  maintainability  and  testability  elements.  While  continuing 
support  of  off-line  testing  RDTSE  is  essential,  more  emphasis  should  be 
placed  on  embedded  testing  support,  which  offers  the  promise  of  simplifying 
the  logistics  pipeline  and  minimizing  the  amount  of  external  test  equipment 
The  era  of  VHSIC  on  the  horizon  necessitates  significant  investments  in 
testing  technology,  prior  to  their  use  in  fielded  systems.  Means  for  pre¬ 
dicting  and  demonstrating  testing  technology  payoffs  are  not  sophisticated 
enough  to  ascertain  their  value  and  to  convince  weapon  systems  designers  of 
their  utility. 


INJECTING  TESTING  TECHNOLOGY  INTO  WEAPON  SYSTEMS  DESIGNS  MUST 


BE  "INSTITUTIONALIZED" 

Institutionalizing  the  injection  of  testing  technology  into  weapon 
system  designs  is  not  being  satisfactorily  accomplished.  Project  managers 
and  their  counterparts  in  industry  are  not  ready  to  risk  involvement  in 
inventing  and  applying  this  technology.  The  analytical  tools,  documenta¬ 
tion,  data  bases,  and  educational  courses  are  not  adequate  t'o  promote 
across-the-board  application  of  testing  technology. 

5.4  THE  MANAGEMENT  OF  TESTING  TECHNOLOGY  REQUIRES  IMPROVEMENT 


The  management  of  testing  technology  is  not  satisfactory  and  is  a 
major  barrier  to  the  success  of  the  program.  Responsibility  is  fractionated 
both  within  OSD  and  within  the  Services.  Over  100  testing  technology  tasks 
with  25  different  sponsors  and  51  performing  activities  supported  by  27 
different  program  elements  are  symptoms  of  the  problem.  The  Services  are 
attempting  to  improve  this  situation  to  the  degree  possible  under  existing 
policy  and  procedures.  Both  the  Navy  and  the  Army  have  established  Testing 
Technology  Strategy  Teams  to  coordinate  and  guide  their  programs.  All  three 
Services  have  central  focal  points  for  coordination  of  testing  technology 
effort,  but  normally  do  not  exert  control  over  the. funding.  This  lack  of 
a  home  for  testing  technology  is  reflected  in  lack  of  support  for  testing 
technology  and  clearly  inhibits  its  transitioning  from  one  RDT&E  category 
to  the  next  and,  subsequently,  its  utilization  in  weapon  systems.  The  funding 
for  testing  technology  is  approximately  50  percent  of  what  is  required.  At 
present,  testing  technology  funding  support  is  much  less  than  1  percent  of 
what  is  being  spent  in  the  testing  area  today.  Industry  IR&D  is  not  aimed 
at  solving  this  problem,  but  yet  is  key  to  solving  the  transitioning  problem; 
and  thus,  must  be  given  additional  incentives,  guidance,  and  controls  to 
make  this  happen. 
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SECTION  6.  RECOMMENDATIONS 


The  following  paragraphs  are  the  major  recommendations  emanating 
from  this  study. 

6.1  INITIATE  A  MAJOR  WEAPON  SYSTEM  DESIGN  TECHNOLOGY  PROGRAM,  WHICH 

INJECTS  TESTING  TECHNOLOGY  INTO  THIS  DESIGN  PROCESS 

A  major  weapon  system  design  technology  must  be  initiated,  which 
injects  testing  technology  into  this  design  process.  Methods  must  be 
developed  for  specifying  mission-driven  testing  requirements  beginning 
with  weapon  system  operational  requirements  and  proceeding  through  the 
weapon  system  acquisition  cycle.  These  requirements  must  be  specified  as 
both  performance  capability  and  "performance  over  time"  parameters.  Tools 
which  can  quantify  the  return  on  investment  for  various  testing  technology 
alternatives  and  permit  trading-off  to  determine  the  proper  mix  of  test 
strategies,  technologies,  and  equipment  must  be  developed.  Measures  of 
effectiveness  to  quantify  the  effect  of  these  mixes  on  operational  readiness 
and  manpower  requirements  are  required.  This  process  must  be  incorporated 
Into  the  weapon  system  computer-aided  design/logistic  support  analysis 
process  to  insure  proper  application.  Design  techniques,  which  promote 
testability,  must  be  developed,  along  with  the  ability  to  predict  and 
demonstrate  testabil i ty  quantitatively. 


6.2  INVEST  IN  EXPANDING  THE  TESTING  TECHNOLOGY  BASE  TO  PROVIDE 


'OFF-THE-SHELF"  PROVEN  ALTERNATIVES  FOR  USE  IN  WEAPON  SYSTEM 


DESIGN 

The  testing  technology  base  needs  to  be  expanded  to  provide  Government 
and  industry  project  managers  with  "off-the-shelf"  proven  alternatives  for 
use  in  their  designs.  Embedded  test  support  should  be  emphasized  including: 

a.  Development  of  non-el ectronic  monitoring  systems  and 
diagnostic/prognostic  techniques. 


b.  Development  of  system-level  (end-to-end)  testing  techniques, 
coupled  with  operational  training  procedures  as  a  means  for 
automating  effective  maintenance,  reducing  manual  testing, 
and  providing  on-the-job  training. 


c.  Development  of  performance  monitoring  hardware  and  software 
to  provide  command  with  an  information  tool  for  ascertaining 
the  readiness  of  his  weapon  systems. 

Support  of  the  three  Service  off-1 tne  ATE  programs  (MATE,  CSS,  and 
ATSS)  should  continue,  as  an  example  of  successfully  transitioning  testing 
technology  to  advanced  and  engineering  development,  and  subsequent  applica¬ 
tion  to  weapon  systems. 

A  formal  integrated  Diagnostics  program  with  a  goal  of  100  percent 
planned  fault  detection  and  fault  isolation  is  required.  The  present  Air 
Force  and  Navy  emphasis  on  this  concept  should  be  further  expanded  and 
adopted  by  the  other  Services.  This  concept  is  supported  by  the  recommenda¬ 
tions  emanating  from  the  OSD  BIT  Workshop  and  the  NSIA  Integrated  Diagnostics 
Conference,  and  thus  has  both  the  Service  and  industry  recognition.  Issuance 
of  formal  OSD  and  Service  policy  is  required,  along  with  auditing  procedures 
to  insure  proper  implementation.  In  addition,  RDTSE  is  required  for  develop¬ 
ment  of  BIT  technology  including  "smart  BIT",  pin  electronics,  etc.  The 
comprehensive  research  and  development  in  maintenance  aiding  should  continue 
with  emphasis  placed  on  implementing  this  technology  as  an  integral  part  of 
Integrated  Diagnostics.  Procedures  to  promote  diagnostic  consistency  from 
factory  testing  through  all  maintenance  and  training  levels  should  be  developed. 

The  testing  of  advanced  devices  such  as  VfSIC,  bubble  memories, 
charge-coupled  devices,  etc.,  should  be  addressed  prior  to  being  incorporated 
in  weapon  system  designs.  Calibration  techniques  for  both  manual  and  auto¬ 
matic  testing  equipment  need  to  be  developed  to  lessen  the  calibration  load 
and  reduce  calibration  costs. 

Lastly,  the  test  bed  concept,  supported  by  a  significant  advanced 
development  effort,  needs  to  be  formalized  as  a  means  for  synergi stical 1 y 
demonstrating  and  integrating  test  technology. 

6>3  INSTITUTIONALIZING  THE  TRANSITIONING  AND  UTILIZATION  OF  TESTING 

TECHNOLOGY 

To  institutionalize  transitioning  and  use  of  testing  technology, 
it  is  required  that: 
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a. 


The  series  of  guidance  documents,  standards,  specifications 
and  handbooks  listed  in  Table  4-2  of  the  basic  report  must 
be  modified  or  prepared,  as  appropriate,  for  use  in  the 
weapon  system  acquisition  process. 

b.  The  testing  technology  data  bases  presently  available  for 
use  are  not  adequate.  Data  is  required  as  an  input  to 
Py,  return  on  investment,  and  testability  prediction 
models.  Data  to  estimate  the  payoffs  from  investments 

in  technology  is  required.  Testability  feedback  of  field 
data  is  required  as  a  means  for  updating  mission-driven 
testing  requirements. 

c.  The  present  Service-  and  industry-offered  courses  in 
automatic  testing  acquisition  and  design  for  testability 
need  to  be  expanded.  In  addition,  a  course  of  ATLAS 
(IEEE  Std.  716)  is  required. 

6.4  INITIATE  A  SERIES  OF  ACTIONS  TO  IMPROVE  TESTING  TECHNOLOGY 

MANAGEMENT 

The  following  is  a  series  of  required  actions  to  improve  the  develop¬ 
ment  and  application  of  testing  technology: 

a.  Current  management  of  testing  technology  RDT&E  is 

fractionated.  A  single  managerial  network  of  testing 
technology  advocates  is  required  extending  from  OSD 
through  the  individual  military  Services.  A  single 
Service  manager  is  not  required,  but  rather  a  series 
of  focal  points  beginning  at  OSD  and  extending  through 
the  l-west  managerial  levels  in  the  Services,  each 
with  appropriate  control  of  funding.  These  focal  points 
not  only  should  be  charged  with  the  responsibility  for 
the  testing  technology  program  for  their  organization, 
but  also  have  appropriate  implementation  authority  to 
assure  proper  application  of  this  technology.  They 
should  be  charged  with  "sign-off"  authority  at  appro¬ 
priate  design  review  points. 
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b.  Integration  of  testing  technology  into  an  overall  logistics 
RDT&E  program  is  required.  Fewer  program  elements, 
improved  integration  of  logistic  effort  supported  within 
these  program  elements  and  an  established  schedule  for 
development  and  transitioning  this  technology  are  integral 
parts  of  this  process. 

c.  DOD  Directives  and  Instructions  5000.1,  5000.2,  and  5000.39, 
etc.,  and  the  Service  implementing  instructions  and  regula¬ 
tions  should  be  reviewed  to  assure  adequate  attention  is 
paid  to  testing  technology.  In  particular,  testability, 

as  a  rigorous  design  discipline,  should  be  injected  into 
these  policy  documents.  Each  Service  and  OSD  should  be 
charged  with,  this  review  responsibility,  with  a  rigorous 
schedule  established  for  modification  and  preparation  of 
appropriate  policy  documents. 

d.  A  program  needs  to  be  established  and  funded  to  identify 
weapon  system  "bad  actors"  and  take  action  to  improve 
the  reliability  and  maintainability  of  these  units. 

Periodic  reports  should  be  prepared  on  the  progress  being 
made. 

e.  Finally,  improved  incentives  for  IR&D  in  testing  technology 
are  required.  Credit  for  IR&D  effort  into  proposals  should 
be  recognized  when  evaluating  these  contractors'  proposals. 
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FOREWORD 


The  recognition  of  the  urgent  need  for  a  strong  DOD  Testing 
Technology  Program  is  not  a  new  idea.  In  a  study,  initiated  by  the 
Assistant  Secretary  of  the  Navy  for  Research  and  Development  in  1976, 
a  Testing  Technology  Program  was  defined.  The  Navy,  through  the  Naval 
Ocean  Systems  Center,  put  "meat  on  the  bones"  and,  together  with  the 
Navy  laboratories  and  other  concerned  activities,  prepared  a  Testing 
Technology  Program  Plan.  Through  the  JLC  Panel  On  Automatic  Testing, 
the  Program  was  expanded  to  include  the  needs  of  the  other  Services. 

Progress  in  implementing  this  Program  has  been  slow.  The 
enthusiasm  of  the  testing  technology  advocates  throughout  DOD  and 
industry  remains  strong.  Unfortunately,  to  a  large  extent,  this 
enthusiasm  has  not  been  effectively  transferred  to  the  many  layers  of 
management  extending  from  DOD  through  Congress. 

The  significant  potential  for  improving  the  reliability  and 
maintainability  by  investments  in  testing  technology  still  remains. 

It  is  anticipated  that,  through  this  report,  the  requirement  for  a 
strong  Testing  Technology  Program  is  "laid  to  rest"  and  support  from 
Congress,  DOD,  and  the  Services  ultimately  will  provide  a  significant 
and  balanced  Program. 

The  recommendations  in  this  report  provide  the  roadmap  for 
reaching  this  goal.  Sound  leadership  is  required  to  make  this 
happen. 

George  W.  Neumann 

Chairman,  Testing  Technology  Committee 
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SECTION  1.  TESTING  TECHNOLOGY 


1.1  INTRODUCTION 

This  study  report  addresses  the  requirements  for  a  testing 
technology  development  program.  The  study  is  part  of  a  larger  Reliability 
and  Maintainability  Improvement  Study  Program.  The  first  portion  of  this 
report  describes  this  entire  study  and  how  testing  technology  fits  into 
its  framework.  This  is  followed  by  a  description  of  the  problem,  scope, 
goals,  objectives,  approach,  content,  payoffs,  conclusions,  and  recommenda¬ 
tions  relating  to  a  testing  technology  program. 

1.2  DESCRIPTION  OF  ENTIRE  RELIABILITY  AND  MAINTAINABILITY  STUDY  PROGRAM 


The  Under  Secretary  of  Defense  has  initiated  a  Joint  OSD-Service- 
Industry  study  for  improving  weapon  system  reliability  and  maintainability 
(R&M).  The  objective  of  the  study  is  to  identify  and  provide  support  for 
high  payoff  actions  which  DOD  can  take  to  improve  the  military  system 
design,  development  and  support  process  in  the  areas  of  reliability  and 
maintainability,  through  innovative  uses  of  advancing  technology  and  program 
structure.  The  study  approach  is  to: 


o  Select,  analyze  and  review  existing  successful  programs 
o  Analyze  and  review  the  related  new  and  advancing  technology 
o  Analyze  and  integrate  review  results 
o  Develop,  coordinate  and  refine  new  concepts 
o  Present  new  concepts  to  DOD  with  recommendations  for 
implementation. 


As  shown  in  Figure  1-1,  this  Joint  OSD-Servi ce- I ndustry  study 
program  is  divided  into  three  distinct  parts.  The  first  part  is  program 
case  studies  to  develop  a  credible  list  of  engineering,  design,  test  and 
contracting  activities,  which  when  followed  will  satisfy  the  study  objec¬ 
tives.  The  second  part  deals  with  effectiveness  analysis  in  order  to 
quantify  the  impact  of  R&M  investment.  The  third  part  addresses  new 
technologies  that  could  lead  to  quantum  improvements  in  R&M  and  readiness. 


This  report  addresses  the  Testing  Technology  portion  of  this  third 
part  of  the  entire  Reliability  and  Maintainability  study.  Thus,  Testing 
Technology  is  a  sub-subset  of  New  Technologies. 

1.3  DESCRIPTION  OF  THE  TESTING  TECHNOLOGY 

Testing  technology  covers  a  gamut  of  research  and  development, 
ranging  from  basic  research  (RDT&E  category  6.1)  to  engineering  development 
(RDTsE  category  6.4).  The  technology  embraces  all  weapon  system  testing 
needs  (e.g.  ,,  electroni cs ,  avionics,  propulsion,  machinery)  related  to 
maintenance  of  those  systems.  As  shown  in  Figure  1-2,  it  includes  test 
equipment;  and  the  logistic  support  of  the  equipment,  which  encompasses  two 
very  expensive  items  -  test  program  sets  and  the  calibration  of  the  test 
equipment  itself.  Embedded  test  support  includes  bui 1 t- i n-test ,  readiness 
monitoring,  and  system  self-alignment.  Also  included  are  two  technologies 
which  are  inexplicably  tied  to  the  design  of  the  weapon  system.  They  are: 

1)  fault-tolerant  design  techniques,  which  when  used  in  conjunction  with 
bui 1 t- i n-test ,  provide  a  very  powerful  readiness  improvement  tool;  and, 

2)  testability  design  techniques,  which  enhance  the  testing  of  units  and 
systems.  Diagnostic  and  prognostic  techniques  are  an  integral  part  of  both 
test  equipment  and  embedded  test  support. 

The  study  addresses  testing  technology  required  to  maintain  al 1 
types  of  weapon  systems.  It  does  not  include  such  testing  as  conducted 
for  reliability  and  maintainability  assurance.  Development  and  operational 
test  and  evaluation  are  also  excluded.  Although  not  specifically  addressing 
factory  testing  units  during  production,  the  integration  of  factory  and 
field  testing  can  save  significant  production,  quality  assurance,  and 
operational  testing  man  hours. 

1.4  TESTING  PROBLEMS  IN  THE  FIELD 

Testing  technology  is  both  a  readiness  and  a  life  cycle  cost  driver. 
It  has  a  significant  effect  on  most  of  the  other  ILS  elements.  It  has  a 
significant  effect  on  the  combat  readiness  and  operational  availability  of 
weapon  systems  by  decreasing  Mean  Time  To  Repair  and  Mean  Logistics  Delay 
Time.  Used  in  conjunction  with  fault-tolerant  design  techniques,  testing 
technology  can  significantly  increase  the  Mean  Time  Between  Failures. 
However,  there  exist  a  number  of  problems  which  inhibit  the  effective 
application  of  this  technology.  Some  of  these  problems  are  summarized  in 
Figure  1-3,  and  discussed  below. 
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TESTING  TECHNOLOGY 


Figure  1-2.  Testing  Technology  Defined 


THE  PROBLEM? 


Weapon  System  Testabint 


Weapon  systems  and  their  units,  which  have  not  been  designed  to  be 
efficiently  tested,  create  excessive  test  times,  cause  excessive  diagnostic 
and  test  programming  costs,  and  require  manpower  and  skill  levels  which  are 
not  readily  available.  Test  times  for  units  presently  fielded  which  are 
designed  to  today's  state  of  the  art,  can  run  from  a  matter  of  minutes  to 
a  number  of  hours.  Figure  1-4  is  a  comparison  of  the  test  requirements  for 
currently  available  units  to  that  required  in  the  immediate  future.  In 
this  case,  a  unit  designed  with  VHSIC  devices  is  used.  It  is  anticipated 
that  maximum  operating  frequencies  for  these  units  will  increase  by  an 
order  of  magnitude;  the  vector  depth  (memory  per  pin)  will  increase  by 
three  orders  of  magnitude;  and,  using  the  same  testing  technology,  the 
overall  test  times  will  increase  by  three  orders  of  magnitude.  Test  pro¬ 
gram  sets  for  testing  a  complex  unit,  such  as  a  "black  box"  removed  from 
an  aircraft,  now  can  cost  over  two  million  dollars  each.  Test  program 
set  costs  for  units  built  with  VHSIC  devices  could  make  the  testing  of 
these  units  impracticable.  This  lack  of  designing  testable  weapon  systems 
and  units  has  forced  the  military  into  multi-level  maintenance  concepts, 
which  require  skilled  technicians  at  each  maintenance  level,  has  lowered 
repair  productivity,  and  has  resulted  in  the  need  for  extra  spares. 

1.4.2  Built-In-Test  (BIT) 

Bui  1 1- in-test  for  weapon  systems,  which  is  being  introduced  into 
the  field  today,  is  not  meeting  diagnostic  specifications.  The  Air  Force 
Test  and  Evaluation  Center  (AFTEC)  in  a  study  of  three  aircraft  concluded 
that  the  diagnostic  capability  as  seen  by  the  user  was  in  the  range  of  50 
to  75. percent  and  that  the  false  alarm  rate  exceeded  o5  percent  in  some 
instances.  In  a  study  conducted  by  the  Naval  Sea  Systems  Command  it  was 
found  that  70  percent  of  the  modules  removed  from  a  weapon  system  were 
eventually  found  to  be  failure-free.  Situations  such  as  these  cause  the 
technicians  to  lose  fa!th  in  the  operation  of  BIT  and  causes  the  logistics 
system  to  operate  inefficiently. 


1.4.3  Test  Equipment 

Historically  the  testing  problem  has  been  satisfied  by  providing 
both  automatic  and  manual  test  equipment  as  the  key  maintenance  and  repair 
tool.  This  has  resulted  in  the  Services  fielding  three  million  units  of 
manual  test  equipment  that  must  be  acquired,  deployed,  and  logistically 
supported.  In  addition,  the  Services  have  procured  and  deployed  approxi¬ 
mately  1,000  different  types  of  ATE's,  many  costing  more  than  a  million 
dollars  each.  Most  of  the  manual  test  equipment  and  many  of  the  building 
blocks,  which  make  up  the  ATE's,  are  commercial  units  which  often  times 
cannot  be  logistically  supported  adequately.  Training  technicians  to  use 
this  wide  variety  of  test  equipment  is  very  difficult  and  time  consuming. 

1.5  PROBLEMS  IN  APPLYING  TESTING  TECHNOLOGY  DURING  THE  WEAPON  SYSTEM 

ACQUISITION  PROCESS 

There  are  a  number  of  problems  in  applying  testing  technology 
during  the  weapon  system  acquisition  process,  which  result  in  excessive 
life  cycle  costs.  Some  of  these  are  described  in  the  following  paragraphs. 

1.5.1  Advancing  Embedded  Testing  Support  Technology 


In  the  past,  the  emphasis  on  satisfying  testing  requirements  has 
been  placed  on  providing  more  and  better  test  equipment.  In  most  cases, 
the  Services  and  industry  have  successfully  achieved  many  advances.  However, 
the  same  emphasis  has  not  been  placed  on  embedded  testing  support.  Embedded 
testing  support  holds  promise  of  reduced  costs  to  achieve  a  given  operational 
availability.  Quite  simply,  creating  the  environment  a  weapon  system  "sees" 
in  operation  is  difficult  and  costly  to  reproduce  in  a  maintenance  shop. 

As  depicted  in  Figure  1-5.  there  is  a  mix  of  embedded  testing  and  test 
equipment  which  will  optimally  satisfy  a  given  operational  availability. 

This  mix  is  not  often  achieved. 

Each  Service  has  a  major  research  and  development  in  test  equipment 
technology.  In  addition,  industry's  IRSD  effort  is  centered  on  test  equip¬ 
ment  hardware  and  software  simply  because  this  is  what  they  can  market. 
Industry  cannot  readily  market  embedded  testing  support  and  so  there  is 
little  IRSD  in  this  area.  Figure  1-6  graphically  depicts  the  effort  which 
more  than  30  companies  sponsor  over  a  hundred  different  projects,  probably 


THE  PROBLEM 


TPS/software  Simplification 
Control  over  ATE  proliferation 
Embedded  support  for  test  equipment 


INDEPENDENT  RESEARCH  &  DEVELOPLMENT 


with  a  dollar  value  of  10  to  20  million  dollars  per  year.  Only  a  fraction 

of  these  dollars  are  spent  on  embedded  testing  support  technology.  Embedded 

« 

testing  support  requires  development  of  "off-the-shelf"  technology,  which  is 
ready  to  apply  at  the  early  stages  of  weapon  systems  design,  as  a  rigorous 
design  discipl ine. 


1.5-2  Applying  Testing  Technology  Early  In  The  Weapon  System 


Acquisition  Cycle 


Historically,  prime  system  designers  have  generally  regarded 
maintenance  (including  testing)  and  logistic  support  as  an  after-design 
concept.  In  addition,  prime  contractors  do  not  adequately  and  properly 
communicate  to  subcontractors  the  scope  and  depth  of  testability  required. 
Neither  is  there  an  adequate  acceptance  test  program  to  evaluate  the  degree 
of  subcontractor  testability  conformance.  As  a  result,  the  Services  have 
been  playing  catch-up  while  many  of  their  weapon  systems  are  down.  Testing 
technology  must  be  an  integral  part  of  weapon  system  design.  As  depicted 
in  Figure  1-7,  this  technology  is  closely  tied  to  computer-aided  design, 
logistic  support  analysis  and  the  automation  of  diagnostics,  test  and 
maintenance.  Without  this  "front-end"  attention,  investments  in  testing 
technology  can  provide  only  marginal  returns  in  readiness  and  cost  reduction. 


1.5- 3  Transitioning  Testing  Technology 

Traditionally,  transitioning  technology  in  any  field  has  been 
difficult.  Transitioning  technology  from  basic  research  to  exploratory 
development  to  advanced  development  and  to  engineering  development  causes 
problems,  because  often  each  of  these  RDTsE  categories  is  managed  by  a 
separate  organizational  entity.  For  the  same  reason,  it  has  proved  diffi¬ 
cult  to  transition  testing  technology  to  weapon  system  design. 

1 . 5 -  ^  Utilizing  Industry  IRSO 

Industry's  I R&D  investment  in  testing  technology  is  significant. 

OSD  and  the  Services  have  emphasized  the  need  for  additional  I R&D  in  each 
one  of  the  logistic  support  elements.  Without  better  incentives,  improved 
coordination,  and  follow-on  visibility,  much  of  this  industry  effort  remains 
unused. 
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SECTION  2.  STUDY  GOAL  AND  OBJECTIVES 

The  objectives  of  this  Testing  Technology  Study  are  to: 

o  Identify  the  required  technology  development. 

o  Estimate  the  impacts  of  thest  technology  devel opments. 

o  Identify  the  key  management  actions  required  to  support 
the  development  and  application  of  this  technology. 

o  Prepare  the  detailed  analysis  necessary  to  justify  and 
defend  the  priorities  that  must  be  afforded  this 
technology  and  the  expected  payoffs. 
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SECTION  3-  DEFINING  THE  REQUIREMENT 


Over  the  past  few  years  the  Services  and  industry  have  taken  a 
number  of  significant  steps  toward  defining  the  requirements  for  testing 
technology  . 

The  Navy's  program  was  established  as  a  result  of  a  study  directed 
by  the  Assistant  Secretary  of  the  Navy  for  Research  and  Development 
1976-  This  study  was  culminated  in  a  "Report  On  Navy  Issues  Concerning 
Automatic  Test,  Monitoring,  And  Diagnostic  Systems  And  Equipment",.  This 
Report  identified  20  basic  Fleet  problems  in  automatic  testing  and  proposed 
14  solutions.  The  Navy's  Testing  Technology  Program  today  is  based  on  the 
findings  of  this  study. 

Two  or  three  years  later,  the  Air  Force  initiated  the  Modular 
ATE  (MATE)  Program.  This  was  a  major  concept  definition  program  in 
competition  between  Sperry  and  Westinghouse.  One  of  the  outputs  of  this 
conceptual  effort  was  a  set  of  MATE  guides  dealing  with  virtually  every 
aspect  of  automatic  testing  and  testability.  These  guides  are  being  used 
by  the  Air  Force  in  the  acquisition  of  their  automatic  testing  hardware  and 
software. 

The  Army  has  just  concluded,  within  the  past  year,. a  DATAT1  study, 
which  resulted  in  22  findings  which  addressed  all  aspects  of  test, 
measurement,  and  diagnostic  equipment.  This  study  has  formed  the  basis  for 
the  Army's  technological  and  managerial  approach  to  solving  testing  problems. 

In  1981  the  Assistant  Secretary  of  Defense,  Manpower,  Reserve 
Affairs  and  Logistics  sponsored  a  Bui  It- In-Test  Equipment  Requirements 
Workshop.  This  Workshop  was  held  for  the  purpose  of  assessing  progress 
and  problems  in  specifying  and  evaluating  built-in-test  used  in  complex 
electronic  equipment.  A  number  of  significant  recommendations  resulted 
from  this  Workshop.  These  recommendations  are  documented  in  the  Institute 
for  Defense  Analysis  Paper,  P-1600.  Fourteen  of  these  recommendations 
dealt  with  specifying  and  evaluating  diagnostics,  including  bui 1 t- in-test . 
Another  set  of  recommendations  were  made,  which  clearly  identified  the 
need  for  technology  development  in  built-in-test  and  diagnostic  techniques. 

This  study  formed  the  basis  for  a  DOD-wide  program  to  improve  built-in¬ 
test  and  diagnostics. 

1  Department  of  the  Army  Test,  Measurement  and  Diagnostic  Equipment  Action  Team. 
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Defining  and  coordinating  testing  technology  effort  among  the 

Services  is  being  accomplished  through  the  framework  of  the  Joint  Logistics 

Commanders  (JLC).  The  JLC  Panel  On  Automatic  Testing  was  formed  in  1978 

and  coordinates  and  guides  the  Joint  Services  Automatic  Testing  program. 

One  of  the  useful  testing  technology  assessments  was  developed  under  this 

Joint  Service  Automatic  Testing  program.  it  assesses  future  testing 
2 

technology  needs  .  This  report  evaluates  the  impact  of  new  technologies  on 
testing  technology  requirements.  It  also  determines  the  applicability  of 
these  new  technologies  to  solving  test  problems.  The  report  covers  new 
technology  in  systems,  components,  electromagnetic  transmissions,  computers, 
electro-optics ,  and  acoustics. 

Testability,  as  a  defined  discipline,  has  been  in  being  for  the 
last  five  or  six  years.  However,  the  institutionalization  of  testability, 
including  the  ability  to  invoke  testability  requirements  in  our  weapon 
systems  designs  to  assure  that  requirements  are  met,  is  in  its  infancy. 
Through  a  Bui  1 t- In-Test/Testabi 1 i ty  Improvement  Program,  initiated 
under  the  Joint  Service  program,  the  path  for  institutionalizing 
testability  through  a  series  of  standardization  documents,  which  are  closely 
tied  to  the  logistics  support  analysis  process,  has  been  defined.  In 
addition,  R&D  has  been  recommended  to  develop,  modify,  and  evaluate  a  series 
of  testability  analytical  tools  to  aid  designers  in  performihg  testability 
trade-offs. 

On  the  other  hand,  industry  has  played  an  important  part  in  defining 
the  Services'  testing  technology  program.  Two  comprehensive  studies  have 
been  supported  by  five  industry  associations:  The  Aerospace  Industries 
Association;  The  Electronic  Industries  Association;  The  National  Security 
Industrial  Association;  The  Shipbuilders  Council  Of  America;  and,  The 
American  Electronics  Association.  The  first  of  these  studies  culminated 
in  a  "Report  Of  Industry  Ad  Hoc  Automatic  Test  Equipment  Project  For  The 
Navy".  This  study  was  directed  almost  totally  at  defining  RDT&E  needs 


2  NOSC  TD  426,  Technology  Assessment,  1980,  Forecast  Of  Future  Test 
Technology  Requirements  (March,  1981). 


in  testing  technology  and  the  institutionalization  of  their  use.  The 
following  areas  were  addressed: 

Software 

Automatic  Test  Generation 
Design  For  Testability 

Propulsion,  Electrical,  and  Auxiliary  Systems  Monitoring 
New  Technology 

Education,  Training,  and  Management 
Advanced  ATE  Concepts 
Operational  Readiness  Monitoring. 

A  similar  type  project  for  the  Joint  Services  was  subsequently  undertaken 
by  these  five  industrial  organizations.  The  Final  Report  of  this  "Industry/ 
Joint  Services  Automatic  Test  Project"  addressed  the  following  11  areas: 

1.  Organizations,  People,  and  Funding 

2.  Military  Equipment  Testab  ility 

3.  Specifications,  Directives,  Controls,  and  Deliverables 

4.  Non-Electronic  Test  Development 

5.  Test  Program  Set  Development  and . Management 

6.  Automatic  Test  Technology  Development 

7.  Data  Banks  and  Models  For  Life  Cycle  Costing,  Logistics 
Support  Analysis,  and  Technology  Assessment 

8.  System- Software  Development  and  Maintenance 

9-  Metrology  and  Calibration 

10.  Training 

11.  Maintenance  Shop  Productivity. 

Because  of  the  close  working  relationship  between  the  Services  and  industry 
during  this  period,  the  Services'  testing  technology  programs  are  totally 
compatible  with  the  industry  recommendations. 


SECTION  4.  TESTING  TECHNOLOGY  REQUIREMENTS 


Investing  in  testing  technology  does  little  good  if  the  Services 
do  not  provide  a  means  for  weapon  system  designers  to  easily  use  this 
technology.  Figure  4- 1  indicates  a  three-pronged  approach  to  this  problem. 
It  includes:  1)  development  of  the  technology  itself,  coupled  with;  2)  the 
tools  used  to  apply  this  technology  in  the  weapon  system  acquisition  process 
and,  3)  appropriate  management  attention  to  ensure  proper  utilization.  Each 
of  these  three  items  is  addressed  in  this  Section* 

4.1  TESTING  TECHNOLOGY 

Figure  4-2  depicts  a  composition  of  required  testing  technology. 

This  structure  includes  the  need  for  weapon  system  testability  design 
techniques.  All  of  these  revolve  around  the  concept  of  a  test  bed  supported 
with  advanced  development  funds  to  evaluate  combinations  of  testing  technol¬ 
ogies,  while  being  able  to  cicertain  the  synergistic  effects  of  each.  This 
test  bed  concept  may  also  be  utilized  in  relation  to  the  other  reliability 
and  maintainability  technologies. 

The  detailed  structure  of  the  Program  is  as  follows: 

A.  Weapon  System  Testability  Design  Techniques 

(1)  CAD/T  Design  Tools 

(2)  LSA  Process 

(3)  T  Prediction  &  Demonstration 

(4)  BIT/MTE/ATE/FOMS 

(5)  Fault-Tolerant  Design 

B.  On-Line  Testing 

(1)  Performance  Monitoring 

(2)  Bui  It- In-Test 

(3)  Maintenance  Aids 

(4)  Non-El ectron ic  Monitoring 

C.  Off-Line  Testing 

(1)  ATE 

(2)  Applications  Software 

(3)  Automatic  Test  Program  Generation  (ATPG) 

(4)  Metrology/Calibration 


Figure  *»-l.  Automatic  Testing  Program 
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Test  Techniques 

(1)  Diagnostics/Prognostics 

(2)  Advanced  Device  Testing 

(3)  Non-Conventional  Testing 

(4)  System-Level  Testing 

E.  Test  and  Evaluation 

(1)  Test  Bed  Demonstrations 

(2)  Experimental  Demonstrations. 

The  approach  to  defining  the  required  testing  technology  effort 

i  s  to: 

a.  Describe  what  is  required  in  each  part  and  subpart 
(Description) 

b.  Determine  on-going  and  completed  work  which  addresses 
this  requirement  (Status),  and 

c.  Identify  what  additionally  needs  to  be  done 
(Requi rement) . 

Table  4-1  was  prepared  as  a  basis  for  making  these  judgments.  The  Table 
summarizes  planned,  on-going,  and  recently  completed  work  in  each  part  of 
the  Program. 

4.1.1  Weapon  System  Testability  Desian  Techniaues 


Testability  is  defined  as  a  design  characteristic  which  allows  the 
status  (operable,  inoperable,  or  degraded)  of  a  unit  (system,  subsystem, 
module,  or  component)  to  be  confidently  determined  in  a  timely  fashion. 
Testability  is  inherently  a  weapon  system  design  issue.  At  present,  the 
use  of  computerized  tools  in  the  design  of  a  weapon  system  is  not  an 
integrated  process.  The  design  of  the  weapon  system  itself  is  part  of  the 
computer-aided  design  (CAD)  process.  Logistics  support  analysis  (LSA)  for 
1LS  should  support  this  CAD  process,  with  testability  as  a  major  driver. 
However,  testability  as  a  rigorous  design  technique  is  in  its  infancy. 
Means  for  specifying,  predicting,  and  demonstrating  weapon  system  testabil 
i  ty  are  not  mature. 


Table  <l  I  Summary  Of  Testing  Technology  Tasks 
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Tasting  (Continued)  Table  H-1.  Summary  Of  Testing  Technology  Tasks  (Continued) 


Summary  Of  Testing  Technology  Tasks  (Continued) 


The  reliability  of  deployed  weapon  systems  has  not  proved  satis¬ 
factory.  Traditional  reliability  approaches  are  expensive,  time  consuming, 
and  not  altogether  satisfactory.  Fault-tolerant  design  techniques  mainly 
have  centered  around  restructuring  at  the  equipment  level,  which  is  costly 
and  creates  a  greater  maintenance  workload.  Present  effort  in  development 
of  fault-tolerant  design  techniques  is  fractionated  with  little  thought  on 
"institutionalizing"  its  use. 

4. 1.1.1  Computer-Aided  Design/Testability  Design  Tools 

Description 

During  the  past  few  years,  the  use  of  computerized  tools  to  design 
a  weapon  system  has  become  commonplace.  Testability  considerations  must  be 
injected  into  these  computer-aided  design  techniques  to  achieve  supportable 

i 

systems.  In  the  device  area,  testability  must  be  a  major  consideration  in 
the  design  of  complex  integrated  circuits. 

Status 

At  least  ten  tasks,  aimed  at  injecting  testability  into  the  CAD 
process  at  the  system,  module,  and  device  level,  have  been  undertaken.  A 
classic  example  of  how  this  can  be  accomplished  is  in  the  VHSIC  Program. 
Recognizing  that  these  complex  devices  could  never  be  successfully  tested 
unless  they  are  designed  to  be  testable,  several  million  dollars  have  been 
spent  in  developing  testable  design. 

Requ?  rement 

The  ability  to  inject  testability  into  computer-aided  design  has 
not  been  institutionalized.  A  major  effort  is  required  to  interface 
testability  with  computer-aided  design  techniques.  The  output  should  be  a 
CAD/LSA/testabi 1 i ty  interface  guide.  In  the  device  area,  additional  funds 
should  be  injected  into  the  VHSIC  Program  to  provide  for  dynamic  testing 
(and  improved  fault  coverage. 

4. 1 . 1 .2  LSA  Process 

Description 

Testability  considerations  must  be  injected  into  the  weapon  system 
LSA  process  as  a  means  for  addressing  external  trade-offs  with  other  logistic 
elements  and  concepts  during  weapon  system  design. 


Status 

All  three  Services  are  addressing  this  problem.  The  Navy  has  just 
completed  a  report  on  the  first  phase  of  their  Bu i 1 t- I n-Test/Testab i 1 i ty 
Improvement  Program.  This  study  analyzes  the  ISA  process,  identifies  methods 
for  specifying  and  evaluating  B I T/testab i 1 i ty  ,  identifies  a  number  of  useful 
testability  analysis  tools  and  techniques,  and  arrives  at  a  recommended 
document  structure  to  institutionalize  the  application  of  BIT  and  testability 
Effort  in  fiscal  year  1983  entails  the  initiation  of  a  testability  analysis 
handbook  and  the  modification  of  a  number  of  currently  available  guides, 
standards,  and  policy  documents.  Rome  Air  Development  Center  has  sponsored 
a  number  of  mission-related  and  log i st i cs- rel ated  trade-off  techniques.  The 
Army  is  working  on  an  LSA  techniques  guide.  Testability  must  be  injected 
into  this  guide. 

Requi remen t 

Although  much  is  being  done  by  the  Services  to  inject  testability 
considerations  into  the  LSA  process,  much  more  is  required.  This  remains  a 
difficult  job,  because  the  LSA  process  itself  is  not  structured  in  such  a 
way  to  permit  straightforward  trade-offs  among  various  logistic  elements. 
Maintenance  strategies  must  be  reviewed  to  assure  that  test  and  repair  are 
done  properly  and  effectively.  The  Navy's  project  to  develop  a  testability 
analysis  handbook  for  making  a  variety  of  types  of  LSA/testab i 1 i ty  trade¬ 
offs  is  underway,  but  is  underfunded  with  very  little  effort  in  the  devel¬ 
opment  of  these  type  analytical  tools.  As  indicated  in  a  GAO  report’, 
measures  of  effectiveness  must  be  developed  to  provide  weapon  system 
designers  a  method  for  evaluating  the  injection  of  testability  alternatives 
into  their  system  design. 

A. 1 . 1 . 3  Testability  Prediction  and  Demonstration 

Descript i on 

Methods  for  predicting  and  demonstrating  how  testable  a  system 
unit  or  device  is,  are  required  so  that  testability  can  be  invoked  in 
procurement  specifications. 


3  GAO  Report  (MASAD-82-38)  of  August  b,  1982. 


Status 


Much  has  been  done  by  the  Services  to  develop  Testability  Figures 
Of  Merit  (T  FOMS)  in  order  that  testability  can  be  predicted  and  evaluated. 

A  survey  on  the  "Application  Of  Testability  Figures  Of  Merit  To  The  Electronic 
System  Acquisition  Process"  was  completed  by  the  Navy  in  May,  T S3 T .  This 
study  identified  and  described  a  number  of  available  means  for  calculating 
T  FOMS,  but  concluded  that  the  best  of  these  T  FOMS  should  be  evaluated 
against  actual  designs  of  units  before  proceeding  with  further  development. 

Requi remen t 

Before  embarking  on  the  program  to  further  develop  T  FOMS,  a 
hardware  evaluation  program  should  be  initiated  so  that  the  effectiveness  of 
the  various  methods  can  be  evaluated  and  further  developmental  work  identified. 
A  testability  analysis  handbook  is  required  to  give  weapon  system  designers 
instruction  on  the  use  of  prediction  tools.  MIL  STD  471  requires  a  revision 
as  a  means  for  demonstrating  testability  in  contractual  terms.  Measures  of 
effectiveness  for  various  T  alternatives  must  be  developed,  so  that  weapon 
system  designers  have  a  means  for  predicting  T  payoffs. 

4. 1.1.1*  BIT/MTE/ATE/FOMS 

Description 

BIT/MTE/ATE/FOMS  are  needed  to  make  a  variety  of  trade-offs 
involving  the  type  BIT,  manual  test  equipment  (MTE) ,  and  ATE  best  suited 
for  a  specific  job;  trade-offs  between  BIT,  MTE,  and  ATE  to  arrive  at  the 
optimum  combination;  and,  injection  of  these  FOMS  into  the  maintainability 
and  reliability  prediction  and  analysis  process. 

Status 

RADC  has  sponsored  the  development  of  a  number  of  trade-off  tools 
in  this  area. 

Requi remen t 

A  means  must  be  developed  for  injecting  these  Figures  Of  Merit  into 
the  ISA  process  in  the  weapon  system  design  process.  Revision  of 
MIL  STD  1513  required. 
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4. 1 . 1 .5  Fault-Tolerant  Design 


Descri ption 

Fault-tolerant  design  includes  a  combination  of  redundancy,  system 
reconfiguration,  and  performance  monitoring  to  achieve  both  weapon  system 
reliability  and  maintainability  goals. 

Status 

Much  effort  has  been  sponsored  in  development  of  a  number  of  system, 
subsystem,  and  device  fault-tolerant  design  techniques  mostly  in  the  basic 
research  and  exploratory  ■development  areas.  NOSC  has  defined  a  Fault- 
Tolerant  Design  Program,  aimed  at  a  combination  of  redundancy,  performance 
monitoring,  and  system  reconfiguration. 

Requi remen t 

A  major  Fault-Tolerant  Design  Program  is  required,  which  includes 
a  combination  of  redundancy,  system  reconfiguration,  and  performance 
monitoring  to  achieve  reliability  and  maintainability  goals.  The  applica¬ 
tion  of  these  techniques  needs  to  be  institutionalized  by  preparing  a 
fault-tolerant  design  guide. 

4.1.2  On-Line  Testing 

On-line  testing  is  defined  as  testing  a  weapon  system  or  unit  in 
its  operational  environment.  It  includes  bu i 1 1- i n-test ,  bui 1 1- i n-test 
equipment,  performance  monitoring,  status  monitoring,  maintenance  aiding, 
etc.  Whether  on-line  testing  is  at  the  ship  level,  the  aircraft  level,  the 
vehicle  level  or  the  weapon  system  level,  it  involves  "des i gn i ng- i n"  a 
comprehensive  testing  hardware  and  software  capability  during  the  acquisition 
process. 

4. 1.2.1  Performance  Monitoring 

Descri pt i on 

Performance  monitoring  is  aimed  at  automatically  ascertaining  the 
"health"  of  a  weapon  system  and  the  environmental  conditions  that  a  weapon 
system  and  the  people  who  operate  and  maintain  it  "see".  The  data  gathered 
is  more  command/operation  oriented,  as  opposed  to  maintenance.  Aboard 
ship,  it  would  supplant  much  of  the  manual  data  recording  and  reporting  now 
used,  thus  improving  the  communication  of  combat  readiness  information  to 
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ill  affected  command  and  maintenance  stations,  and  reduce  the  possibility 
of  ma i n tenance- i nduced  failures  while  performing  planned  maintenance.  For 
aircraft,  recording  of  in-flight  data  can  be  used  for  subsequent  diagnostic 
and  prognostic  purposes. 

Status 

A  major  program  for  development  of  a  shipboard  Operational  Readiness 
Monitoring  capability  transitioned  from  exploratory  development  and  was, 
subsequently,  cancelled  by  Congress i ona I  budget  action.  A  demonstration 
test  bed  was  developed  and  the  feasibility  and  worth  of  the  concept  proven. 

Requ i rement 

The  project  should  be  revitalized  and  proceed  through  advanced 
development,  so  that  requirement  can  be  included  in  ship  developmental 
specifications.  Revision  of  MIL  STD  1326  is  required  to  control  interfaces 
with  weapon  systems. 

A. 1 .2.2  Built-in-Test  (BIT) 

Description 

Bui  1 1- in-test  is  integral  to  the  design  of  the  platform,  vehicle, 
system,  subsystem,  or  device  requiring  testing.  BIT  at  the  system  and 
subsystem  level  fault  detects  and  fault  isolates  down  to  a  given  ambiguity 
group.  BIT  at  the  device  level  provides  for  the  ready  testing  of  the 
device  to  ascertain  i ts_  operat i ng  condition. 


Status 

The  Air  Force  Test  and  Evaluation  Center  conducted  a  comprehensive 
analysis  on  the  effectiveness  of  BIT  on  three  major  aircraft.  A  major 
program  to  improve  the  effectiveness  of  BIT  has  been  undertaken  by  the 
Navy,  with  a  Fleet  survey  of  BIT  effectiveness  for  two  dozen  deployed 
weapon  systems.  Two  BIT  workshops  have  been  conducted.  All  of  these 
actions  have  helped  define  needed  improvements  in  BIT.  A  major  policy 
and  documentation  review  is  presently  underway,  so  that  BIT  application  is 
properly  institutionalized. 


Requi rement 

RDT&E  is  required  to  provide  off-the-shelf  technology  for  weapon 
system  projects.  This  includes: 

a.  Development  of  "Pin  Electronics"  concepts  for  BIT 

b.  Development  of  "smart  BIT"  using  artificial  intelligence 
and  knowledge-based  systems 

c.  Further  development  of  BIT  for  VHSIC  devices. 

4. 1.2. 3  Maintenance  Aids 

Description 

Maintenance  aids,  sometimes  referred  to  as  job  performance  aids, 
provide  diagnostic  and  procedural  information  to  assist  technicians  in 
maintaining  weapon  systems.  It  is  possible  to  embed  much  of  this  capability 
into  the  weapon  system  design  and  thus  enhance  the  on-line  testing 
capabi 1 i ty. 

Status 

A  major  survey  of  maintenance  aids  was  conducted  in  FY82.  Significant 
R&D  has  and  is  being  sponsored  to  develop  maintenance  aids,  including: 

a.  PEAM  Program  -  Development  of  devices  for  the  dual 

purpose  of  maintenance  aiding  and  on-the-job  training. 


b.  EPIC  Program  -  Development  of  maintenance  aids  as  a 
means  for  providing  relatively  untrained  technicians 
with  valuable  assistance,  thus  delaying  formal 
technician  training  until  individual  capability  and 
motivation  can  be  evaluated  and  re-enlistment 
intentions  are  known. 


c.  HTIPS  -  A  major  C.3  program  designed  to  provide 

electronic  delivery  of  operational,  maintenance,  and 
logistic  information  to  the  Fleet.  Maintenance  aiding 
is  one  facet  of  the  program. 


All  these  programs  are  presently  in  their  evaluation  phase. 


a.  Development  of  LCC  models  to  determine  the  degree  of 
application  of  maintenance  aids. 

b.  Determination  of  the  appropriate  mix  of  BIT,  ATE,  and 
maintenance  aids. 

c.  Institutionalizing  their  application. 

d.  Development  of  the  BIT/ATE  guided-probe  concept. 

e.  Standardization  of  hardware  and  software. 
k. 1.2. k  Non-Elect ronic  Monitoring 

Description 

RDTsE  for  the  testing  and  monitoring  of  non-electronic  weapon 
system  and  equipment  (e.g.,  propulsion,  electrical,  auxiliary  machinery) 
is  required. 

Status 

a.  Automatic  test  systems  to  analyze  performance  and 
operation  of  diesel  and  gas  turbine  engines  have  been 
developed  and  put  into  operation  by  the  Navy  test 
community-.  These  test  systems  have  increased  the 
operational  availability  by  reducing  maintenance- 
related  documents  of  the  prime  systems  using  these 
engines. 

b.  The  National  Bureau  of  Standards  has  prepared  a 
handbook  describing  available  sensors  and  application 
information  for  non-electronic  monitoring. 

c.  High-accuracy  differential  pressure  transducers  to 
sense  the  fan  discharge  mach  number  for  aircraft 
gas  turbine  engines  have  been  developed. 

Requi remen t 

a.  Development  of  a  shipboard  machinery  performance 
monitoring  system. 


b.  Development  of  a  BIT/testabi 1 i ty  non-electronic 


4.1.3  Off-Line  Testin 


Off-line  testing  is  accomplished  by  a  combination  of  automatic  and 
manual  test  equipment,  coupled  with  the  necessary  software  for  test  program 
sets  required  to  diagnose  faulty  units.  Development  of  manual  test  equip¬ 
ment  required  by  the  Services  is  being  accomplished  by  industry,  using  IRSD 
funds.  Except  in  special  cases,  the  Services  do  not  and  should  not  invest 
their  dollars  to  develop  manual  test  equipment.  On  the  other  hand,  automatic 
test  equipment,  to  a  large  degree,  is  designed  to  the  operational  and  support 
requirements  of  the  Services.  Logistic  support  (including  calibration)  is 
required  for  all  types  of  test  equipment. 

4. 1.3.1  Automatic  Test  Equipment  (ATE) 

Description 

ATE  is  required  to  provide  for  timely  testing  of  units  removed  from 
a  weapon  system  for  repair. 

Status 

Each  Service  is  in  the  process  of  developing  advanced,  mission-oriented 
ATE.  The  Navy  is  in  the  final  phases  of  conceptual  development  of  the  Consoli¬ 
dated  Support  System  (CSS).  This  project  addresses  total  repair  shop  through¬ 
put  (i.e.,  ATE  for  mul t i pie-weapon  support,  the  shop  management  system,  the 
shop  environment,  and  its  logistic  support).  The  Air  Force's  Modular  ATE 
(MATE)  Program  is  presently  being  implemented.  It  standardizes  the  ATE  acqui¬ 
sition  process  and  the  system  architecture  (e.g.,  IEEE  Std.  488  bus,  MIL  STD 
1750  computer  architecture,  IEEE  Std.  716  ATLAS).  The  Army's  Automatic  Test 
Support  System  (ATSS)  development  is  just  underway.  It  is  aimed  at  forward- 
level  deployment  of  ATE. 

Requi remen t 

Both  Navy  and  Army  ATE  programs  are  investing  in  Pin  Electronics  as 
the  next  generation  ATE.  In  essence,  this  is  the  concept  of  "ATE  on  a  chip", 
which  eliminates  the  complex  switching  problem  of  present-day  ATE  by  placing 
a  testing  capability  at  each  pin  of  a  UUT.  In  addition,  more  effort  is  required 
as  a  means  for  addressing  units  employing  VHSIC  devices. 


This  covers  techniques  for  developing  test  program  sets  (TPS's), 
including  test  programs,  interface  devices,  and  associated  documentation. 


Status 

a.  Standardization  of  a  test  language  was  accomplished  by 
working  with  industry  and  ultimately  sponsoring  IEEE  to 
control  and  monitor  the  ATLAS  test  language. 

b.  A  standard  military  subset  of  ATLAS  (IEEE  Std.  716)  was 
produced  to  simplify  the  use  of  this  standard  language. 

c.  Techniques  and  guidelines  were  developed  for  TPS 
configuration  management  and  for  validation  and 
verification  methods.  Experience  has  shown  that 
uncontrolled  development  of  test  program  sets  for  ATE 
can  be  very  costly. 

d.  A  TPS  Acquisition  Guide  was  developed  for  project 
managers  use. 

e.  A  Test  Program  Design  Guide  was  developed  for  use  by 
designers  of  test  programs. 

Requi rement 

TPS  cost  estimating  techniques  must  be  validated  as  a  means  for 
controlling  their  acquisition  costs. 

4. 1.3.3  Automatic  Test  Program  Generation  (ATPG) 

Description 

This  covers  techniques  for  computer-aided  methods  for  generating 
analog,  digital,  and  hybrid  test  programs. 

Status 

a.  Digital  ATPG  techniques  have  been  developed  and  refined 
to  significantly  reduce  costs  of  test  programming. 

b.  A  "Selection  Guide  For  Digital  Test  Program  Generation 
Systems"  (NAVMATP  9493,  CARCOMP  70-9,  AFLC  800-41, 

AFSC  800-41,  NAVMC  2718)  1991  was  produced,  which 
described  29  available  ATPG's. 


Requi rement 

ATPG  for  units  employing  VHSIC-type  devices  must  be  developed. 
Because  of  the  complexity,  standard  techniques  are  unacceptable  for  use. 

4. 1.3.4  Metrology /Cal ibration 

Description 

This  covers  metrology  and  calibration  methods  and  standards  for 
all  types  of  test  equipment. 

Status 

A  number  of  standards  and  equipments  have  been  developed.  One  of 
the  major  developments  has  been  MECCA,  which  is  an  automatic  calibration 
system.  Hundreds  of  these  systems  have  been  deployed  at  both  shore  and 
shipboard  calibration  labs,  at  a  substantial  reduction  in  cost  and  manpower 

Requi rement 

a.  Development  of  standards  are  required,  such  as: 

o  Microwave  Standards  1 8-45  GHz 

o  High  RF  Power  Standards 

o  Electrical  Voltage  Standards 

o  I FF/TACAN/VOR  Standards 

o  Multimeter  Standards 

o  Instrument  Micro-Controllers. 

b.  Techniques  for  calibrating  ATE  at  its  terminals,  in 
lieu  of  calibrating  each  box  (module)  separately,  are 
requi red. 

4.1.4  Test  Techniques 

The  extensive,  and  sometimes  unnecessary,  maintenance  actions  on 
weapon  systems  place  high  demands  on  personnel  and  test  equipment,  and 
adversely  affect  combat  readiness.  Furthermore,  the  employment  of  new 
and  emerging  technologies,  which  offer  opportunities  for  reducing  manning 
requirements  for  future  weapon  system  operation;  will  impose  increased 
demands  on  maintenance  personnel. 
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k. 1.4.1  Oi agnostics/Prognostics 


Diagnostic  and  prognostic  techniques  are  needed  in  both  the 
electronic  and  non-el ectronic  areas  in  order  to  meet  a  planned  requirement 
for  planned  100  percent  fault  detection  and  fault  isolation. 

Status 

For  electronics,  a  large  number  of  diagnostic  techniques  have  been 
developed  ranging  from  computerized  automatic  test  generation  techniques 
to  the  use  of  maintenance  dependency  charting  techniques.  There  is  little 
diagnostic  consistency  during  weapon  system  design,  beginning  with  FMEA 
and  progressing  through  test  requirement  documentation  requirements,  BIT 
design,  ATE  test  program  generation,  maintenance  manuals,  and  maintenance 
training.  Limited  guidance  exists  on  a  variety  of  diagnostic  approaches 
avai lable. 

Research  has  been  performed  in  the  area  of  simulation  and  modeling 
to  aid  in  fault  analysis,  fault  prediction,  TPS  automated  production,  and 
in  the  design  of  testable  digital  architectures. 


Requi rements 


Development  of  methods  for  specifying  diagnostic 
parameters,  which  are  mission  driven. 

Development  of  trade-off  procedures/man-machi ne 
allocation  procedures  to  provide  the  best  mix  of 
diagnostic  hardware  and  software  for  a  given  application 

Review  and  revision  of  applicable  standards,  specifi¬ 
cations  and  guides  to  institutionalize  Integrated 
Diagnostics. 

Development  of  Integrated  Diagnostics  data  base  and 
feedback  systems. 

Development  of  Integrated  Diagnostics  application 
handbook. 

Development  of  diagnostics  and  prognostics  techniques 
for  non-el ectronic  applications. 

Development  of  a  non-electronic  testability  guide. 


k. 1.4. 2  Advanced  Device  Testing  and  Non-Convent i onal  Testing 

Description 

This  covers  a  variety  of  testing  techniques  for  testing  advanced 
devices  or  using  these  advanced  devices  for  non-convent i onal  testing 
appl i cat  ions. 

Status 

A  substantia]  number  of  tasks  have  been  supported  by  the  Services 
dealing  with  the  testing  of  lasers,  optics,  antenna  measurements,  fiber 
optics,  high-power  tubes,  etc.  Examples  are: 

a.  Investigation  of  12  areas  for  improving  the  capability 
of  E/0  airborne  devices. 

b.  Measurements  and  calibration  methods  for  laser  weapon 
systems . 

c.  Methods  for  analysis  of  unique  interference  tests 
for  advanced  optical  systems. 

Requi remen t 

A  number  of  tasks  under  the  JLC  Automatic  Testing  Program  remain 
unfunded.  These  are  in  the  MIS  305  series  and  deal  with  the  testing  of 
memories,  charge-coupled  devices,  E/0  devices,  etc. 

4.1. A. 3  System-Level  Testing 

Description 

This  covers  system- level  testing  (e.g.,  communications,  E.W.)  and 
is  usually  characterized  by  end-to-end  tests  for  system  checkout. 

Status 

Some  communications  system- level  test  techniques  have  been  developed 


Requi remen t 

An  opportunity  exists  to  combine  operational  training  and  system- 
level  testing  for  the  ships  combat  and  damage  control  systems.  This 
capability  can  be  embedded  in  the  weapon  system.  Radars,  fire  control 
systems,  sonars,  etc.,  can  be  stimulated  using  distributed  microprocessors, 
allowing  checkout  of  each  system  based  on  real-life  operational  scenarios. 
On-the-job  operator  training  can  be  performed  as  an  adjust  capability.  PMS 
checks,  system  checkout,  system  alignment,  trend  analysis,  etc.,  all  appear 
feasible. 


4.1.5  Test  And  Evaluation  (Test  Beds) 


Description 


None  of  the  technology  development  discussed  in  the  above  paragraphs 
can  be  developed  in  a  vacuum.  Scientific  test  beds,  including  prototypes, 
need  to  be  utilized  for  use  in  evaluating  various  testing  technology  improve¬ 
ments  in  an  integrated,  realistic  operating  environment. 

Status 

Although  a  number  of  test  beds  have  been  constructed  and  used,  none 
has  across-the-board  application. 

Requi remen t 

Scientific  test  beds,  including  prototypes,  need  to  be  developed 
and  made  available  for  use  in  evaluating  various  testing  technology  improve¬ 
ments  in  an  integrated,  realistic  operating  environment.  The  main  thrust 
of  the  development  effort  should  focus  on  emerging  testing  technology  advances 
in  performance  monitoring,  bui 1 t- in-test,  fault  detection,  fault  diagnostics, 
self  test,  automated  calibration,  self  calibration,  fault-tolerant  design, 
data  buses,  and  all  other  technological  improvements  which  must  be  evaluated 
prior  to  their  inclusion  in  either  the  weapon  system  design  or  an  ATE 
application.  The  importance  of  a  test  bed  is  the  ability  to  measure  the 
synergistic  effects  of  these  various  advances  in  technology,  such  a:.: 

a.  The  operation  of  built-in-test  with  redundant  systems. 

b.  The  compatibility  of  built-in-test  with  off-line  testers. 

c.  The  extent  of  bui 1 t- i n-test  required  with  fault-tolerant 
design. 

d.  The  prediction  and  measurement  of  testability  as  a  design 
parameter. 

e.  A  demonstration  of  the  applicability  and  worth  of  advances 
in  technology  to  a  potential  user  (e.g.,  project  manager, 
field) . 

f.  The  compatibility  of  system-level  testing  and  operator 
training. 

g.  The  worth  of  ope*ational  readiness  monitoring  in  today's 
field  envi ronment. 
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Summarizing  what  is  required: 


a.  Development  and  operation  of  a  test  bed(s). 

b.  Experimentally  applying  this  testing  technology  to 
weapon  systems,  including  comprehensive  tracking  to 
ascertain  the  worth  of  these  concepts. 

4.2  ACQUISITION  TOOLS 

Applying  the  output  of  testing  technology  requires  the  institution¬ 
alization  of  acquisition  tools.  These  include: 

a.  Preparation  of  design  and  application  guides, 
standards  and  specifications  for  weapon  system 
designers. 

b.  Establishment  and  maintenance  of  informational 
data  banks  for  use  with  analytical  models. 

c.  Development  and  offering  of  educational  courses 
for  project  managers  and  weapon  system  designers 
in  the  application  of  tasting  technology. 

Table  4-2  summarizes  mandatory  and  non-mandatory  documents  and 
educational  courses,  which  are  required  to  accomplish  institutionalization 
and  application  of  testing  technology  in  relation  to  the  program  develop¬ 
mental  requirements.  The  status  of  each  is  indicated.  Much  effort  is 
required  to  prepare  or  modify  these  documents. 

4.3  MANAGEMENT 

Transitioning  test  technology  to  weapon  system  design  requires  a 
number  of  management  initiatives. 

a.  An  organizational  entity  is  required  within  each 
Service  and  OSD  to  plan,  coordinate  and  transition 
testing  technology  through  to  weapon  system  design. 

b.  Policy  directives  are  required  for  each  Service  to 
establish  such  an  organization  and  to  assure  technology 
developments  are  funded  and  pursued. 

c.  Controls  over  development  of  testing  technology  and 
its  application  are  required. 

d.  Methods  for  Joint  Service  coordination  of  testing 
technology  is  required. 
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PROGRAM  SUMMARY 


4.4 

Table  4-1  summarizes  recently  completed,  on-going,  and  planned 
technology  development.  It  is  a  reasonably  complete  picture  of  the  program. 
Over  100  tasks  have  been  identified,  and  an  equal  number  of  IR&D  efforts  is 
estimated.  Figure  4-3  is  a  list  of  25  known  sponsors  and  Figure  4-4  lists 
51  known  performing  activities  drawn  from  Table  4-1.  Figure  4-5  is  a  list 
of  27-Program  Elements,  which  support  this  effort. 

4.4.1  Program  Funding 

Table  4-3  summar i zes  Testing  Technology  requirements  and  status,  and 
estimates  the  available  and  required  funding  ranges.  The  basic  research 
and  exploratory  development  funded  effort  estimates  are  reasonably  accurate, 
because  they  are  characterized  by  almost  level -funding.  Advanced  and 
engineering  development  funding  varies  sharply  year-by-year,  and  thus,  what 
is  shown  is  a  "snapshot".  Figure  4-6  is  a  funding  summary,  which  indicates 
that  basic  research  and  exploratory  development  is  funded  at  $7M  annually  - 
50  percent  of  requirements.  Advanced  and  engineering  developments  are 
funded  at  $27M  -  77  percent  of  requirements.  This  does  not  include  a 
substantial  deficit  in  the  CSS  out-year  funding. 

4.4.2  Program  Priori  ties 

Figure  4-7  prioritizes  the  various  parts  of  the  Testing  Technology 
Program  based  on  test  issues;  the  affect  on  weapon  system  operational 
readiness,  life  cycle  cost  and  manpower  considerations;  technical  risk;  and 
the  size  of  the  funding  deficiency.  High  priorities  are  given  to: 

o  Weapon  system  design,  using  testab i 1 i ty/B I T/fau 1 t 
tolerance/performance  monitoring  techniques, 
incorporated  into  the  CAD/LSA  process. 

o  Diagnost i c/prognost ' c  techniques,  integrating  FMEA, 
B I T/Testab i l i ty  maintenance  aiding,  ATPG  into  a 
cohesive,  institutionalized  process. 

o  Non-e 1 ect ron i c  test  and  monitoring  techniques. 
System-level  test  techniques. 


o 


Figure  *4-3.  Sponsors  Of  Testing  Technology 
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Figure  4-4.  Performing  Activities 
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Figure  k-S-  Testing  Technology  Program  Elements 


6.3  PROGRAM  FOR  VALIDATION 
REVISION  OF  LSA  TECHNIQUES 
HANDBOOK 

REVISION  OF  MIL  STD  1388 


REVISION  OF  MIL  STD  1513 


PROGRAM) 


GUIDE 

TURBINE  AND  DIESEL  MONITORING 
EFFORT 


MAJOR  PROGRAM  FOR  ATE  CALIBRATION 
TECHNIQUES 


NON-ELECTRON I C  APPLICATIONS 
DEVELOP  MACHINERY  DIAGNOSTIC 
PROCEDURES 


Figure  4-6.  Annual  Funding  Summary  ($M) 
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Testing  Technology  has  a  great  many  interfaces  with  other  R&M 
technologies.  Figure  4-8  is  a  Venn  diagram  which  illustrates  some  of 
these  interfaces  and  reflects  quantitatively  how  much  of  these  technologies 
are  common.  The  major  point  is  that  none  of  the  R&M  technologies  can 
adequately  solve  the  readiness  problem  by  themselves,  and  a  correct  mix 
of  each  is  required  if  the  problem  is  to  be  solved. 


ig  Technology  Interfaces  With  Other  Technolog 


SECTION  5.  BENEFITS  ANALYSIS  AND  TECHNOLOGY  PAYOFFS  OF  TESTING  TECHNOLOGY 


Thus  far,  this  report  has  detailed  the  current  programs  In  testing 
technology  and  Identified  the  required  additional  programs.  The  judgment 
on  what  is  required,  based  upon  expert  opinion,  has  been  documented  in 
major  reports  published  since  1975-  Despite  this  level  of  credibility, 
the  specific  benefits  and  payoffs  of  testing  technology  must  be  proven. 

The  measures  of  effectiveness  defined  earlier  In  this  report  show 
many  specific  areas  where  benefits  could  be  quantitif led.  In  fact,  there 
are  several  dozen  measures  of  effectiveness  which  relate  testing  technology 
to  logistics  parameters.  These  parameters  are  further  related  to  reliabil¬ 
ity  and  maintainability  measures  which  impact  system  readiness.  The  problem 
in  benefits  analysis  is  really  a  problem  of  selecting  among  the  measures, 
or  stated  in  other  terms,  which  payback  is  of  most  value.  Each  of  the  bene¬ 
fits  from  each  of  the  testing  technology  research  areas  must  be  traded-off 
against  each  other.  The  value  of  a  particular  benefit,  therefore,  becomes 
subjective  and  in  competition  with  other  benefits. 

5.1  COMMON  DENOMINATOR  IS  COST  REDUCTION 

There  is,  however,  a  common  denominator  in  valuing  the  testing 
technology  trade-offs.  Since  readiness  requires  improvements  In  reliability 
and  maintainability,  and  since  these  improvements  can  be  achieved  through 
many  different  approaches,  the  only  common  denominator  becomes  cost  reduc¬ 
tion.  Therefore,  the  overriding  benefit  of  testing  technology  must  be 
measured,  not  only  in  its  impact  on  readiness  but  also  on  its  Impact  on 
reducing  total  ownership  cost  of  a  system. 

5.2  OPERATIONAL  REQUIREMENTS  MUST  CALL  OUT  PERFORMANCE  VS.  TIME 

Before  trade-off  of  various  approaches  can  begin,  it  is  imperative 
that  the  requirements  for  iogistlcally  driven  measures  and  concepts  be  in 
the  specifications  for  new  systems.  This  requirement  must  be  inputted  at 
the  earliest  levels  (Conceptual  Phase)  of  a  system  life  cycle.  Operational 
requirements  specified  in  early  documents  such  as  Statement  of  Need  must 
state  the  required  logistic  related  time  dependencies.  If  these  parameters 
are  not  included  In  the  Operational  Statement  of  Requirements,  then  the 


trade-offs  are  unlikely  to  be  required  downstream.  Usually  project  managers 
consider  logistics  as  a  follow-on  cost  and  find  that  in  a  near-term  budqet 
problem  the  logistics  cost  can,  and  should,  be  deferred.  If  these  time- 
related  requirements  are  properly  stated  uo-front  as  real  needs,  they  can 
be  assured  of  staying  in  a  program.  This  report  stronqly  suggests  in  its 
conclusion  that  the  performance  capability  of  a  system  be  supplemented 
with  time-dependent  measures  of  performance  capability.  In  this  manner, 
the  time  dependence  will  drive  the  logistics  requirements  and  the  project 
managers  will  become  aware  of  the  importance  of  logistics  on  system  per¬ 
formance  over  time.  This  is  referred  to  in  this  report  as  "speaking  the 
language"  of  the  weapon  system  designer. 

5.3  TRADE-OFFS  DRIVEN  BY  COST 

Given  an  operational  requirement  which  drives  the  logistics  side 
of  a  system,  the  benefits  trade-offs  will  be  between  such  items  as: 

o  Mean  Time  Between  Failure 

o  Mean  Time  to  Repair 

o  Mean  Logistics  Delay  Time 

o  Personnel  Numbers 

o  Personnel  Skills 

-  Training 

-  Maintenance  Aiding 

o  Spares/Facilities/Space 

o  Built-in-Test  vs.  Off-Line  Testing 

o  Organizational,  Intermediate  and  Depot  Level  Test  Allocations 

o  Acquisition  Cost/Performance  Capability 

o  Mobility 

o  Environment. 

It  is  apparent  that  on  a  life  cycle  cost  basis,  one  of  the  key 
drivers  Is  the  acquisition  cost  of  the  system.  The  relative  importance 
of  acquisition  cost  makes  the  trade-offs  extremely  complicated.  This  is 
because  the  acquisition  costs  have  been  escalating  due  to  Increasing  system 
complexity  relating  to  the  threat.  In  addition,  the  Administration 
has  a  major  program  underway  to  build  up  the  strength  of  conventional  and 
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strategic  U.  S.  Forces.  The  importance  of  acquisition  costs,  therefore, 
cannot  be  understated.  To  do  so  Mould  completely  undermine  the  purpose 
of  this  study. 

it  must  be  concluded  then  that  the  overriding  benefit  that  testing 
technology  must  address  is  cost  reduction.  Payoffs,  which  can  be  measured 
in  total  cost  reduction,  must  be  prioritized  so  that  they  can  be  used  to 
balance  the  spiralling  acquisition  cost  in  Defense  spending. 

5.**  PRODUCTIVITY  IS  KEY  TO  COST  REDUCTION 

When  viewed  in  these  terms,  it  becomes  very  obvious  that  the  key 
to  testing  technology  benefits  analysis  can  be  measured  in  terms  to  pro¬ 
ductivity.  Productivity  improvement  must  be  a  product  of  technology 
investment. 

5.*».1  Japanese  Model 

The  model  usually  used  to  measure  productivity  is  the  Japanese 
model.  The  Japanese  model  is  based  upon  a  concerted  effort  to  provide  a 
major  investment  in  automation  across  all  Japanese  industry.  Within  the 
Japanese  model,  the  dollars  required  for  this  major  investment  come  from 
such  items  as: 

* ♦  Inventory  Reduction  During  The  Manufacturing  Process. 

This  is  achieved  by  very  tight  scheduling  of  workflow 
utilizing  automation  techniques. 

k •  Reduced  Production  Cost  Through  Quality  Improvement. 
Quality  Circles  and  Quality  Programs  lead  to  a  much 
higher  yield  in  production  lines  than  the  equivalent 
U.  S.  factories. 

c’  Intensive  Value  Engineering  Programs.  Japanese  manu¬ 
facturers  are  utilizing  Value  Engineering  within  their 
own  operations  and  at  all  subcontractor  and  vendor  levels. 
Cost  reductipn  goals  are  set  and  Value  Engineering  programs 
instituted  on  a  continuing  basis  to  reduce  product  cost. 
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d.  Encouraged  Savings  By  Employees  In  Company  and  Private 

Savings  Accounts.  It  is  not  illegal  in  Japan  for  companies 
to  utilize  employee  savings  proqrams  as  a  source  of  direct 
investment  in  automation  equipment.  This  is  equivalent  to 
U.  S.  corporations  utilizing  pension  plans  for  capital 
investment ,  which  is  illegal  in  the  U.  S. 

in  order  to  direct  the  flow  of  money  from  the  above  areas  into 
automation,  the  Japanese  model  is  driven  by  deliberate  government  coopera¬ 
tion  with  industry.  A  very  broad  proqram  of  tax  incentives  encourages 
the  flow  of  money  into  automation  projects.  Japanese  workers  receive  tax 
benefits  for  savings.  In  addition  to  government  support,  the  Japanese 
model  is  driven  by  an  extremely  favorable  attitude  of  employees.  This 
attitude  is  principally  the  result  of  the  Japanese  work  culture.  Japanese 
companies  have  a  very  paternal  attitude  toward  employees.  Employees  are 
not  in  danger  of  layoffs  during  cost  cutting  operations.  In  addition,  an 
intense  feeling  of  pride  in  product  and  quality  Is  engendered  among 
employees  on  a  continuing  basis.  These  key  drivers  are  contrary  to  the 
U.  S.  culture.  Therefore,  attempts  to  duplicate  the  Japanese  model  on 
a  one-for-one  basis  will  not  work. 

5. *.2  Return  On  Investment  Drives  U.  S.  Decisions 

The  key  U.  S.  productivity  improvement,  therefore,  must  be  driven 
by  technology  without  the  benefit  of  an  outpouring  of  dollars,  as  is  the 
case  in  the  Japanese  economy.  American  investment  must  be  based  upon  a 
calculable  return  on  investment,  which  makes  the  capital  expenditure 
justifiable  on  a  short-term  basis.  Short-term  Is  measured  in  a  period  of 
three  to  five  years.  An  opportunity  exists  in  testing  technology  to  affect 
such  a  formula  for  investment.  This  opportunity  is  available  because  most 
of  the  problems,  which  are  impacted  by  low  yield  and  poor  quality,  are 
embodied  in  the  "rework  cost"  of  production  defects  and  "repair  costs"  of 
field  failures  and  returns. 

In  essence,  reducing  the  cost  of  rework  and  repair  is  equivalent 
to  Increasing  the  quality  and  yield  of  a  product. 


5.5  DIAGNOSTICS  DRIVES  COST  REDUCTION 

The  driver  in  rework  cost  and  repair  cost  is  not  *  he  physical  repair 
actions,  but  the  cost  of  diagnostics.  Diagnostics  is  the  capability  of 
determining  what  exactly  is  wrong  when  a  system  fails. 

Simply  stated,  diagnostics  addresses  the  following  two  questions: 

Am  I  OK? 

If  not,  why  not? 

If  testing  technology  can  reduce  ambiguity  group  size  in  diagnostics, 
then  the  basic  cost  drivers,  namely  repair,  rework,  and  spares  cost,  will 
be  cut  sharply. 

This  report  addresses  overall  testing  technology  improvement  with 
emphasis  on  weapon  system  design,  on-line  testing,  and  off-line  testing. 
Specific  diagnostic  improvements  are  derived  from  product  design,  production 
testing  In  our  factories,  and  service  testing  by  the  military  user. 

5.6  SPECIFIC  BENEFITS  AND  PAYOFFS 

Figure  5“1  graphically  depicts  the  escalating  OsMN1*  cost  for  component 
repair  for  Navy  aviation.  These  figures  are  from  studies  conducted  by  the 
Navy's  Aviation  Supply  Office. and  communicate  a  three-fold  increase  in  the 
cost  of  component  repair  over  a  nine-year  period.  Figures  5-2  through  5"5 
indicate  that  a  small  number  of  systems  contribute  to  the  major  portion  of 
the  maintenance  cost.  Figure  5-6  indicates  that  for  a  rather  nominal  'nvest- 
ment  these  key  items  could  be  made  more  reliable  and  maintainable.  Improve¬ 
ments  in  239  key  items  on  just  four  aircraft  could  reduce  the  total  0&MN  and 
aviation  component  repair  cost  by  30  percent  -  a  saving  of  $331.6  million 
dollars  In  1987.  Multiplying  this  figure  by  the  number  of  years  these  com¬ 
ponents  are  required  can  amount  to  many  billions  of  dollars. 

0  course,  there  are  varieties  of  methods  for  obtaining  and  improving 
the  reliability  and  maintainability  of  these  components.  However,  improving 
the  diagnostics  and  testing  for  these  components  is  a  major  cost  driver. 

These  figures,  although  only  applicable  to  four  Navy  aircraft,  have  a 
significant  impact  on  testing  technology  payoffs  In  two  ways.  The  first 
Is  embedded  test  support.  If  these  items  had  been  designed  to  be  testable 
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Figure  5-2.  A-6E/EA-6B  Repair  Costs 


TOTAL  ANNUAL  O&MN  COST  PECULIAR  TO  S-3A  APPROXIMATES  (FY  82)  $49.6M 
ELEVEN  KEY  SYSTEMS  (20%)  REPRESENT  74%  OF  COST 
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Figure  5"3-  S-3A  Repair  Costs 
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Figure  5“5-  FA-18  Repair  Costs 


»  FOR  COMPONENT  REPAIR 


Figure  5~6.  OsMM  $  For  Component  Repair 


at  the  time  the  weapon  system  was  designed,  there  would  be  little  need  for 
after-the-fact  improvements.  Secondly,  if  a  diagnostic  capability  through 
improved  test  program  sets  for  existing  off-line  ATE  was  added,  much  the 
same  reduction  in  cost  would  occur.  The  difference  in  these  two  approaches 
is  that  the  second  alternative  requires  significantly  more  dollars  to 
obtain  much  the  same  cost  savings.  Expanding  on  this  example,  the  specific 
benefits  and  payoffs  from  embedded  test  support  and  off-line  testing  are 
projected  in  the  following  paragraphs. 

5.6.1  Embedded  Test  Support 

5. 6. 1.1  Maintenance  Benefits 

Designing- in  testability  on  new  weapon  systems  has  the  highest 
potential  of  all  approaches  in  testing  technology  research  and  development. 
A  10  to  20  percent  reduction  in  life  cycle  costs  can  be  achieved  by 
employing  good  testability  design. 

The  principal  technological  payback  from  embedded  testing  is  the 
improvement  in  system  readiness.  Readiness  is  improved  by  enhancing  the 
capability  of  the  system  to  pinpoint  the  areas  where  problems  may  exist 
(diagnostics).  On-line  testing  will  help  focus  for  the  operator  the 
particular  areas  that  require  attention  by  maintenance  crews.  Unambiguous 
monitoring  of  system  performance  capability  for  critical  parameters  and 
the  measurement  of  the  time  dependence  of  that  capability  is  essential  for 
system  operation.  By  adding  embedded  testing  capability,  we  enhance  both 
the  performance  and  the  support  of  that  system. 

The  cost  of  embedded  testing  capability  approximates  10  percent 
of  the  total  system  development  cost.  This  cost  includes  both  design 
cost  and  fabrication  of  the  on-line  testing  hardware  and  software. 

Industry  estimates  show  that  this  cost  is  recovered  prior  to  delivery 
in  cost  savings  during  factory  production  testing. 

Effectiveness  is  measured  by  the  ability  of  the  on-line  testing 
system  to  detect  faults  to  a  leVel  which  matches  the  support  concept 
(resource  allocation)  for  that  system. 


Performance  monitoring  and  diagnostic  capability  can  now  be 
achieved  with  modern,  sophisticated  electronics.  Also,  usinq  new  tech¬ 
nology,  false  alarms  can  be  drastically  reduced  by  usinq  microprocessor 
controlled  systems  which  become  intelligent  built-in  test  sensors. 

Technology  advances,  which  are  directly  related  to  improving  the 
capability  for  on-line  testing  and  monitoring,  include  RSD  programs  such 
as  Universal  Pin  Electronics.  Utilizing  advanced  semiconductor  techniques 
and  devices  such  as  high-speed  analog-to-digi tal  and  digi tal -to-analog  con¬ 
verters  one  can  develop  functional  digital,  parametric  digital,  and  analog 
test  capabilities  within  a  single  measurement  channel.  This  channel  can 
be  reduced  in  size  ("ATE  on  a  chip")  using  hybrid  techniques.  Availability 
of  this  technology  to  the  industry  at  large  will  result  in  the  more  effec¬ 
tive  usage  of  on-line  testing.  Technological  advances  in  effect  can  be 
used  to  actually  reduce  operational  and  maintenance  complexity  of  a  system 
as  seen  by  the  user. 

Parameters  which  may  be  used  to  express  the  payoff  relate  to  P^. 

PT  is  a  measure  of  the  time  dependence  of  a  performance  capability.  For 
example,  using  this  measure  one  can  monitor  degraded  levels  of  performance. 
By  monitoring  performance  levels,  systems  will  not  be  shut  down  when  small 
anomalies  change  the  performance  characteristics.  Without  performance 
monitoring,  systems  must  be  shut  down  since  the  operator  is  not  aware  of 
what  his  reduced  capability  is.  Also,  by  reducing  ambiguity  group  size 
(exact  location  of  failure)  through  on-line  testing,  the  performance  time 
dependency  on  logistic  support  will  be  enhanced. 

The  expected  payoff  from  on-line  testing  is  projected  in  Figure  5*7 

It  is  expected  that  the  addition  of  on-line  testing  will  not  create 
any  degradation  in  system  performance.  In  fact,  without  on-line  testing, 
the  user  tends  to  use  what  is  coimonly  referred  to  as  an  "Easter  egging" 
for  changing  parts  which  he  feeis  may  be  bad.  By  constant  reshifting  of 
parts  in  and  out  of  the  system  without  the  benefit  of  testing,  the  tech¬ 
nician  is  actually  degrading  the  system.  Therefore,  accurate  on-line 
testing  will  reduce  system  degradation  potential. 


Trade-offs  between  payoff  and  degradation  is  certainly  positive. 
The  issue  is  the  ability  to  provide  the  additional  on-line  capability 
without  impacting  system  size/weight,  and  within  an  affordable  cost. 

It  is  expected  that  the  recent  advances  in  technology  make  on-line 
testing  feasible  at  this  time.  Failure  to  apply  this  technology  will 
result  in  complex  fielded  systems  with  ail  the  problems  which  relate 
to  that  complexity  (particularly  manpower  problems). 

Paybacks  of  on-line  testing  take  advantage  of  the  fact  that  the 
system  in  its  operational  state  Is  capable  of  self-analysis  using  internal 
electronics.  Similar  measurements  attempted  off-line  require  that  the 
environment  of  the  system  be  recreated  at  great  expense  in  the  test  equip¬ 
ment  consoles. 


5. 6. 1.2  Productlvi t\ 


movement  and  Factory  Testing  Benefits 


The  benefits  and  payoffs  of  diagnostic  improvement  in  factory 
testing  are  quantifiable  in  terms  of  "return  on  investment".  In  view  of 
this,  most  U.  S.  factories  have  made  significant  investments  in  automation 
leading  to  improve  diagnostics.  Specifically,  in-circuit  testing  which 
provides  100  percent  diagnostics  (for  catastrophic  failures)  has  resulted 
in  increased  purchases  for  that  type  of  equipment,  which  approaches  200 
million  dollars  since  1980.  Most  U.  S.  corporations,  which  had  purchased 
automatic  test  equipment  using  Government  contract  funds,  have  now  turned 
to  their  own  investment  capital.  Return  on  investment  calculations  show 
returns  of  in  excess  of  25  percent  with  many  returns  of  50  percent  for 
capital  equipment  used  for  testing.  Productivity  gains  in  the  factory 
are  keyed  on  increased  diagnostics  which  reduce  rework  costs.  This 
improvement  in  the  factories  far  exceeds  any  improvement  in  yield  due  to 
quai i ty  circles. 


Since  the  return  on  investment  is  so  explicit,  Government  inter¬ 
vention  is  not  required  to  enjoy  this  gain  in  productivity.  Embedded 
support  concepts,  however,  are  more  difficult  to  quantify  and,  therefore, 
must  be  secured  by  the  Government  through  improved  specifications. 


Service  testing,  conmonfy  referred  to  as  off-line  testing,  is 
usually  done  at  two  or  three  levels  of  maintenance.  These  levels  include 
Organizational,  Intermediate  and  Depot  Levels.  The  ability  to  repair 
fielded  units  at  forward  levels  has  a  direct  impact  on  the  amount  of 
spares  required  to  support  a  specific  readiness  requirement.  Trade-offs 
of  ATE  vs.  spares  has  resulted  in  military  deployment  of  millions  of 
dollars  worth  of  test  equipment  over  the  past  decade.  Based  upon  lessons 
learned  and  recent  studies  of  a  major  industry  group,  the  Joint  Logistics 
Commanders  have  instituted  an  automatic  testing  plan.  A  key  part  of  this 
plan  includes  specific  Services  off-line  testing  programs  such  as  Air 
Force's  MATE  (Modular  Automatic  Test  Equipment),  Navy's  Consolidated 
Support  System  (CSS),  and  the  Army's  Automatic  Test  Support  System  (ATSS) 
Programs.  These  programs  represent  the  first  major  R&D  effort  in  off-line 
ATE  development  by  the  military.  Each  of  these  programs  attacks  a  unique 
set  of  mission-driven  problems.  MATE  is  maturing  rapidly  and  is  being 
institutionalized  across  the  Air  Force.  The  Navy  Consolidated  Support 
System  is  midway  in  a  study  phase  and  will  be  enterinq  Full-Scale  Develop¬ 
ment  in  late  1983.  The  Army  Automatic  Test  Support  System  is  now  entering 
a  study  phase. 

Results  to  date  have  shown  very  specific  cost  benefits.  Due  to 
improved  management  techniques  and  reduction  in  proliferation  of  new 
designs,  the  off-line  ATE  programs  in  the  Air  Force  are  expected  to  save 
at  least  100  million  dollars  per  year.  Navy  studies  in  Consolidated 
Support  System  indicate  that  personnel  savings  up  to  30  percent  will  be 
achieved  and  throughput  will  be  doubled  at  carrier  1-Level. 

5.6.3  Summary  Of  Benefits 

In  summary,  the  combination  of  embedded  test  support  and  off-line 
testing  can  have  a  significant  affect  on  maintenance  costs.  The  reduction 
will  be  primarily  from  reduced  repair  cost  in  the  field.  In  addition, 
rework  cost  in  the  factories  across  the  Nation  is  being  drastically 


reduced  by  improved  diagnostics.  This  dual  impact  of  rework  cost 
reduction  in  the  factory  and  repair  cost  reduction  in  the  field  will 
drive  down  both  the  maintenance  costs  and  the  acquisition  costs  of 
new  systems. 

In  view  of  the  increased  expenditures  for  new  weapons  and  the 
escalating  acquisition  and  ownership  costs  created  by  th i-s  complexity, 
it  is  imperative  that  the  recommendations  in  this  report  be  immediately 
implemented.  The  military  threat  must  be  met  at  "affordable  cost".  The 
consequence  of  increased  Defense  production  at  "any  cost"  could  be 
National  economic  disaster. 


SECTION  6.  CONCLUSIONS 


As  a  result  of  this  study,  the  following  conclusions  have  been 

reached. 

6.1  TRADITIONAL  WEAPON  SYSTEM  RELIABILITY  AND  MAINTAINABILITY 
DESIGN  TECHNIQUES  ARE  NO  LONGER  SATISFACTORY 

Traditional  design  techniques  for  injecting  reliability  and  main¬ 
tainability  technology  into  weapon  systems  are  no  longer  satisfactory. 
Testability  and  testing  requirements  must  be  injected  into  weapon  system 
operational  requirements,  requests  for  proposals,  and  system  specifications 
beginning  at  the  weapon  systems  concept  formulation  stage  and  continuing 
throughout  the  acquisition  cycle.  These  requirements  must  be  specified 
as  "design  requirements"  and  measurable  over  the  acquisition  cycle  of  the 
weapon  system.  To  accomplish  this,  we  must  learn  to  "speak  the  language" 
of  the  weapon  system  designer.  A  "performance  over  time"  concept  must 
replace  the  "supportabil Ity"  concept,  with  "performance  over  time"  equal 
in  importance  to  performance  capability.  Effectiveness  must  have  the 
relationship  between  performance  capability  and  "performance  over  time": 

Et»Pc  X  PT. 

To  do  this,  we  must  learn  how  to  specify  Py.  It  must  be  mission-driven  and 
relatable  to  acquisition  and  ownership  costs. 

6.2  IMPROVEMENT  IN  THE  TECHNOLOGY  BASE  IS  REQUIRED 

The  present  technology  base  does  not  exist  to  significantly  improve 
this  situation.  Tools  do  not  exist  to  integrate  and  trade-off  various 
reliability,  maintainability  and  testability  elements.  While  continuing 
support  of  off-line  testing  RDT&E  is  essential,  more  emphasis  should  be 
placed  on  embedded  testing  support,  which  offers  the  promise  of  simplifying 
the  logistics  pipeline  and  minimizing  the  amount  of  external  test  equipment 
The  era  of  VHSiC  on  the  horizon  necessitates  significant  investments  in 
testing  technology,  prior  to  their  use  in  fielded  systems.  Means  for  pre¬ 
dicting  and  demonstrating  testing  technology  payoffs  are  not  sophisticated 
enough  to  ascertain  their  value  and  to  convince  weapon  systems  designers  of 
their  uti I i ty. 


6.3 


INJECTING  TESTING  TECHNOLOGY  INTO  WEAPON  SYSTEMS  DESIGNS  MUST 


BE  "INSTITUTIONALIZED11 

Institutionalizing  the  injection  of  testing  technology  into  weapon 
system  designs  is  not  being  satisfactorily  accomplished.  Project  managers 
and  their  counterparts  in  industry  are  not  ready  to  risk  involvement  in 
inventing  and  applying  this  technology.  The  analytical  tools,  documenta¬ 
tion,  data  bases,  and  educational  courses  are  not  adequate  to  promote 
across-the-board  application  of  testing  technology. 

6.4  THE  MANAGEMENT  OF  TESTING  TECHNOLOGY  REQUIRES  IMPROVEMENT 

The  management  of  testing  technology  is  not  satisfactory  and  is  a 
major  barrier  to  the  success  of  the  program.  Responsibility  is  fractionated 
both  within  OSD  and  within  the  Services.  Over  100  testing  technology  tasks 
with  25  different  sponsors  and  51  performing  activities  supported  by  27 
different  program  elements  are  symptoms  of  the  problem.  The  Services  are 
attempting  to  improve  this  situation  to  the  degree  possible  under  existing 
policy  and  procedures.  Both  the  Navy  and  the  Army  have  established  Testing 
Technology  Strategy  Teams  to  coordinate  and  guide  their  programs.  All  three 
Services  have  central  focal  points  for  coordination  of  testing  technology 
effort,  but  normally  do  not  exert  control  over  the  funding.  This  lack  of 
a  home  for  testing  technology  is  reflected  in  lack  of  support  for  testing 
technology  and  clearly  inhibits  its  transitioning  from  one  RDTtE  category 
to  the  next  and,  subsequently,  its  utilization  in  weapon  systems.  The  funding 
for  testing  technology  is  approximately  50  percent  of  what  is  required.  At 
present,  testing  technology  funding  support  is  much  less  than  1  percent  of 
what  is  being  spent  in  the  testing  area  today.  Industry  IR&D  is  not  aimed 
at  solving  this  problem,  but  yet  is  key  to  solving  the  transitioning  problem; 
and  thus,  must  be  given  additional  incentives,  guidance,  and  controls  to 
make  this  happen. 


SECTION  7.  RECOMMENDATIONS 


The  following  paragraphs  are  the  major  recommendations  emanating 
from  this  study. 

7.1  INITIATE  A  MAJOR  WEAPON  SYSTEM  DESIGN  TECHNOLOGY  PROGRAM.  WHICH 
INJECTS  TESTING  TECHNOLOGY  INTO  THIS  DESIGN  PROCESS 

A  major  weapon  system  design  technology  must  be  initiated,  which 
injects  testing  technology  into  this  design  process.  Methods  must  be 
developed  for  specifying  mission-driven  testing  requirements  beginning 
with  weapon  system  operational  requirements  and  proceeding  through  the 
weapon  system  acquisition  cycle.  These  requirements  must  be  specified  as 
both  performance  capability  and  "performance  over  time"  parameters.  Tools 
which  can  quantify  the  return  on  investment  for  various  testing  technology 
alternatives  and  permit  trading-off  to  determine  the  proper  mix  of  test 
strategies,  technologies,  and  equipment  must  be  developed.  Measures  of 
effectiveness  to  quantify  the  effect  of  these  mixes  on  operational  readiness 
and  manpower  requirements  are  required.  This  process  must  be  incorporated 
Into  the  weapon  system  computer-aided  design/logistic  support  analysis 
process  to  insure  proper  application.  Design  techniques,  which  promote 
testability,  must  be  developed,  along  with  the  ability  to  predict  and 
demonstrate  testability  quantitatively. 

7.2  INVEST  IN  EXPANDING  THE  TESTING  TECHNOLOGY  BASE  TO  PROVIDE 
"OFF-THE-SHELF"  PROVEN  ALTERNATIVES  FOR  USE  IN  WEAPON  SYSTEM 
DES IGN 

The  testing  technology  base  needs  to  be  expanded  to  provide  Government 
and  industry  project  managers  with  "off-the-shelf"  proven  alternatives  for 
use  in  their  designs.  Embedded  test  support  should  be  emphasized  including: 

a.  Development  of  non-el ectronic  monitoring  systems  and 
diagnostic/prognostic  techniques. 

b.  Development  of  system-level  (end-to-end)  testing  techniques, 
coupled  with  operational  training  procedures  as  a  means  for 
automating  effective  maintenance,  reducing  manual  testing, 
and  providing  on-the-job  training. 


c.  Development  of  performance  monitoring  hardware  and  software 
to  provide  command  with  an  information  tool  for  ascertaining 
the  readiness  of  his  weapon  systems. 

Support  of  the  three  Service  off-line  ATE  programs  (HATE,  CSS,  and 
ATSS)  should  continue,  as  an  example  of  successfully  transitioning  testing 
technology  to  advanced  and  engineering  development,  and  subsequent  applica¬ 
tion  to  weapon  systems. 

A  formal  Integrated  Diagnostics  program  with  a  goal  of  100  percent 
planned  fault  detection  and  fault  isolation  is  required.  The  present  Air 
Force  and  Navy  emphasis  on  this  concept  should  be  further  expanded  and 
adopted  by  the  other  Services.  This  concept  is  supported  by  the  recommenda¬ 
tions  emanating  from  the  OSD  BIT  Workshop  and  the  NSIA  Integrated  Diagnostics 
Conference,  and  thus  has  both  the  Service  and  industry  recognition.  Issuance 
of  formal  OSD  and  Service  policy  is  required,  along  with  auditing  procedures 
to  Insure  proper  implementation.  In  addition,  RDT&E  is  required  for  develop¬ 
ment  of  BIT  technology  including  "smart  BIT',  pin  electronics,  etc.  The 
comprehensive  research  and  development  in  maintenance  aiding  should  continue 
with  emphasis  placed  on  implementing  this  technology  as  an  integral  part  of 
Integrated  Diagnostics.  Procedures  to  promote  diagnostic  consistency  from 
factory  testing  through  all  maintenance  and  training  levels  should  be  developed. 

The  testing  of  advanced  devices  such  as  VHSIC,  bubble  memories, 
charge-coupled  devices,  etc.,  should  be  addressed  prior  to  being  incorporated 
in  weapon  system  designs.  Calibration  techniques  for  both  manual  and  auto¬ 
matic  testing  equipment  need  to  be  developed  to  lessen  the  calibration  load 
and  reduce  calibration  costs. 

Lastly,  the  test  bed  concept,  supported  by  a  significant  advanced 
development  effort,  needs  to  be  formalized  as  a  means  for  synergi sti cal  1 y 
demonstrating  and  integrating  test  technology. 

7.3  INSTITUTIONALIZING  THE  TRANSITIONING  AND  UTILIZATION  OF  TESTING 

TECHNOLOGY 

To  institutionalize  transitioning  and  use  of  testing  technology, 
it  Is  required  that: 
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a.  The  series  of  guidance  documents,  standards,  specifications 
and  handbooks  listed  in  Table  4-2  must  be  modified  or  pre¬ 
pared,  as  appropriate,  for  use  in  the  weapon  system 
acquisition  process. 

b.  The  testing  technology  data  bases  presently  available  for 
use  are  not  adequate.  Data  is  required  as  an  input  to 
Py,  return  on  investment,  and  testability  prediction 
models.  Data  to  estimate  the  payoffs  from  investments 

in  technology  is  required.  Testability  feedback  of  field 
data  is  required  as  a  means  for  updating  mission-driven 
testing  requi rements . 

c.  The  present  Service-  and  industry-offered  courses  in 
automatic  testing  acquisition  and  design  for  testability 
need  to  be  expanded  as  indicated  in  Table  4-2.  In 
addition,  a  course  on  ATLAS  (IEEE  Std.  716)  is  required. 


INITIATE  A  SERIES  OF  ACTIONS  TO  IMPROVE  TESTING  TECHNOLOGY 
MANAGEMENT 

The  following  is  a  series  of  required  actions  to  improve  the  develop 
ment  and  application  of  testing  technology: 

a.  Current  management  of  testing  technology  RDTsE  is 

fractionated.  A  single  managerial  network  of  testing 
technology  advocates  is  required  extending  from  OSD 
through  the  i 'dividual  military  Services.  A  single 
Service  manager  is  not  required,  but  rather  a  series 
of  focal  points  beginning  at  OSD  and  extending  through 
the  lowest  managerial  levels  in  the  Services,  each 
with  appropriate  control  of  funding.  These  focal  points 
not  only  should  be  charged  with  the  responsibility  for 
the  testing  technology  program  for  their  organization, 
but  also  have  appropriate  implementation  authority  to 
assure  proper  appl icat ion  of  this  technology.  They 
should  be  charged  with  "sign-off"  authority  at  appro- 


b.  Integration  of  testing  technology  into  an  overall  logistics 
RDT&E  program  is  required.  Fewer  program  elements, 
improved  integration  of  logistic  effort  supported  within 
these  program  elements  and  an  established  schedule  for 
development  and  transitioning  this  technology  are  integral 
parts  of  this  process. 

c.  DOD  Directives  and  Instructions  5000.1,  5000.2,  and  5000.39, 
etc.,  and  the  Service  implementing  instructions  and  regula¬ 
tions  should  be  reviewed  to  assure  adequate  attention  is 
paid  to  testing  technology,  in  particular,  testability, 

as  a  rigorous  design  discipline,  should  be  injected  into 
these  policy  documents.  Each  Service  and  OSD  should  be 
charged  with  this  review  responsibility,  with  a  rigorous 
schedule  established  for  modification  and  preparation  of 
appropriate  pol icy  documents. 

d.  A  program  needs  to  be  established  and  funded  to  identify 
weapon  system  "bad  actors"  and  take  action  to  improve 
the  reliability  and  ma fnta inabi I i ty  of  these  units. 

Periodic  reports  should  be  prepared  on  the  progress  being 
made. 

e.  Finally,  improved  incentives  for  IR6D  in  testing  technology 
are  required.  Credit  for  IR&D  effort  into  proposals  should 
be  recognized  when  evaluating  these  contractors'  proposals. 
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